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Abstract 

Selection  of  combat  aircraft  during  crisis  action  planning  can  be  of  critical 
importance.  In  determining  the  value  of  force  mixes,  it  is  proposed  that  one  can  evaluate 
extrinsic  and  intrinsic  value  separately.  Intrinsic  value  is  the  designed  capability  of  a 
weapons  platform  to  accomplish  a  specified  aerospace  mission.  Extrinsic  value  is  the 
expected  appropriateness  of  such  platforms,  given  the  environmental  characteristics  in 
which  they  must  operate.  This  research  develops  a  decision  support  tool  for  planners  in 
determining  the  extrinsic  value  of  force  mixes,  which  then  expedites  the  selection  of  best 
overall  force  mixes. 

The  research  included:  content  analysis  of  official  guidance.  Critical  Decision 
Methodology  interviews,  a  Delphi  study  (to  define  and  quantify  the  factors,  and  establish 
a  hierarchy  and  global  weights),  generation  of  the  Value  Focused  Thinking  decision  tool, 
and  establishment  of  an  appropriate  representation  of  the  relationship  between  extrinsic 
and  intrinsic  value. 

This  research  provides  planners — throughout  the  USAF — with  a  decision  support 
tool  that  objectively  compares  alternative  force  packages  for  specific  deployments.  This 
represents  a  first  step  toward  codifying  or  formalizing  the  art  of  force  selection.  These 
results  will  help  reduce  the  crisis  action  response  timeline,  and  should  lead  to  more 
accurate  modeling  of  force  mix  applicability. 


A  QUANTITATIVE  DECISION  SUPPORT  MODEL  TO  AID  SELECTION  OF 
COMBAT  AIRCRAFT  FORCE  MIXES  FOR  CONTINGENCY  DEPLOYMENT 

I.  Introduction 

Background 

Contingency  deployment  planning  is  as  critical  to  the  United  States  Air  Force’s 
(USAF)  wartime  mission  accomplishment  as  any  task  could  be.  During  both  deliberate 
and  crisis  action  planning  processes,  decision  makers  must  allocate  resources  and 
manpower  with  sufficient  strategic  and  tactical  vision  to  affect  total  mission 
accomplishment  under  conditions  of  both  uncertainty  and  risk.  Limited  resources  of  all 
types  among  all  of  the  armed  forces  significantly  constrain  decision  makers  during  such 
planning,  and  necessitate  paradigm-shifting,  creative  solutions. 

The  USAF  has  responded  with  the  Air  Expeditionary  Force  (AEF)  concept  while 
simultaneously  pursuing  initiatives  to  reduce  the  resources  required  for  deployment  (the 
deployment  footprint)  and  to  increase  management  flexibility  with  regard  to  resource 
allocation  [Godfrey,  1998;  Looney,  1996].  The  AEF  concept  reflects  the  Air  Force’s 
policy  of  Global  Engagement,  in  which  forces  deploy  worldwide  from  the  continental 
United  States  on  short  notice  rather  than  maintaining  a  permanent  presence  at  forward, 
overseas  locations.  Additionally,  the  AEF  approach  attempts  to  provide  operational  units 
important  reconstitution  and  training  periods  by  allowing  for  cyclic  rotation  of  the  Air 
Expeditionary  Wings  (AEW’s)  to  which  the  units  are  attached.  The  AEF  concept  allows 
the  decision  makers  visibility  of  which  units  are  both  on  tap  and  available  for 
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deployment.  It  also  generates  flexibility  in  managing  scarce  resources  by  virtue  of  the 
relative  centrality  of  forces  within  the  continental  United  States,  which  can  then  be 
projected  anywhere  in  the  world. 

Enhancing  the  visibility  and  flexibility — as  well  as  the  response  time — of 
resources  in  today’s  planning  environment  is  vital  to  future  USAF  mission  success.  The 
Defense  Advanced  Research  Projects  Agency  (DARPA)  has  spent  four  years  developing 
a  distributed  computing  architecture  that  will  create  a  new  and  improved  process  for 
deployment  planning  for  all  U.S.  military  forces.  This  architecture,  known  as  the 
Advanced  Logistics  Project  (ALP),  has  been  geared  toward  total,  near  instantaneous  asset 
visibility  among  all  Department  of  Defense  (DoD)  units.  Such  real-time  visibility  will 
naturally  present  planners  with  many  options  but  will  also  present  possibly  complicated 
tradeoffs. 

It  is  the  measure  of  these  potential  tradeoffs  in  resources  and  force  mixes  to 
optimal  mission  accomplishment  that  constitutes  the  “value”  of  a  given  decision.  This  is 
where  the  Air  Force  Institute  of  Technology  (AFIT)  enters.  This  research  effort  is 
focused  in  support  of  AFIT’s  contribution  to  the  ALP  endeavor.  AFIT’s  role  in  this 
project  is  to  provide  a  value  assessment  architecture  that  provides  decision  support  to 
contingency  planners  by  identifying  utility  levels  for  force  mixes  (different  mixes  of 
fighters  and  bombers)  and  their  time-phased  requirements  and  respective  lift  constraints. 
Overall  force  mix  utility  (value)  levels  may  be  determined  through  an  appropriate 
combination  and  valuation  of  factors,  and  those  factors  can  be  divided  into  two  general 
categories:  extrinsic  and  intrinsic.  In  theory,  separate  utilities  can  be  established  for  each 
category.  The  extrinsic  utility,  or  value,  of  a  force  mix  (interchangeably  referred  to  as  an 
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asset  set )  includes  all  factors  that  are  external  to  the  set’s  inherent  operational 
capabilities,  such  as  beddown  location,  fuels  support,  and  political  considerations. 

Conversely,  intrinsic  or  inherent  utility  or  value  is  represented  by  the  actual 
combat  capability  of  a  resource.  Obviously,  the  first  concern  in  evaluating  a  force  mix  is 
that  the  aircraft  comprising  the  set  are  fully  capable  of  carrying  out  the  designated 
aerospace  combat  missions  required  by  the  theater  commander.  Although  intrinsic  value 
is  critical  to  force  mix  selection  and  would  serve  as  a  worthwhile  research  topic,  it  is 
beyond  the  scope  of  this  research. 

Problem  Statement 

Given  that  ALP  will  grant  instantaneous  visibility  of  all  available  resources,  while 
providing  an  architecture  to  generate  a  detailed  deployment  plan  in  just  minutes  versus 
hours  or  days,  planners  at  all  appropriate  echelons  will  have  an  opportunity  to  judge 
many  deployment  scenarios  at  the  touch  of  a  computer  key.  Planners  will  have 
expediting  tools  and  complete  resource  visibility  never  before  seen. 

With  such  information  at  their  fingertips,  how  then  would  campaign  planners 
choose  from  among  the  competing  deployment  plans?  This  research  intends  to  answer 
that  question. 

As  mentioned,  ALP  will  provide  the  visibility  and  the  architecture.  AFIT’s 
specific  contribution  to  ALP  will  be  the  “M-R  VAT,”  the  Mission-Resource  Value 
Assessment  Tool.  The  M-R  VAT  will  provide  support  for  time-phased  decisions  that 
goes  beyond  current  time  phased  force  deployment  documents  (TPFDD’s)  and  will  help 
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to  evaluate  operational  campaign  requirements  along  a  timeline  before  the  deployment 
occurs.  Chapter  II  provides  more  information  on  the  ALP  and  M-R  VAT. 

The  basic  premise  of  both  M-R  VAT  and  this  supporting  research  is  that 
campaign  planners  should  identify,  strategically  and  then  tactically,  the  time-phased 
sortie  requirements  by  specific  mission  type — before  the  resource  selection  process 
begins.  That  way,  a  truer  representation  of  what  is  needed,  where  it  is  needed,  and 
exactly  when  it  is  needed,  will  surface.  Only  then  can  fully-informed  decisions  be  made 
as  to  which  force  mixes  are  most  appropriate  and  present  the  best  value. 

To  aid  decision  makers  in  selection  of  force  mixes,  this  research  proposes  a 
decision  model,  or  tool,  that  integrates  and  quantifies  all  of  the  extrinsic  factors  that  affect 
force  mix  selection.  It  is  important  to  note  that  campaign  planners  will  judge  the  inherent 
operational  value  of  force  mixes  (their  absolute  mission  capability)  separate  from  this 
model,  using  criteria  that  are  beyond  the  scope  of  this  research.  This  research  provides 
one  necessary  step  to  aid  war  planners  in  selecting  the  highest  value  force  mixes  in  the 
shortest  possible  time. 

However,  in  accomplishing  the  goal  of  this  research,  it  is  necessary  to  establish 
that  both  extrinsic  and  intrinsic  factors  exist  and  to  clearly  define  each  type.  Chapter  II 
addresses  these  issues. 

Research  Questions 

Central  to  providing  an  executable,  integrative  decision  analysis  model  to  enhance 
the  force  mix  selection  efforts  of  campaign  planners,  this  research  addresses  the 
following  six  questions: 
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1 .  Can  the  factors  affecting  force  mix  selection  in  response  to  theater  crises  be  clearly 
categorized  as  extrinsic  and  intrinsic,  and  can  these  categories  be  clearly  defined? 

2.  What  extrinsic  factors  are  important  to  campaign  planners  when  selecting  aircraft 
force  mixes  in  response  to  theater  crises? 

3.  How  are  these  factors  quantified,  and  what  are  the  relationships  between  them? 

4.  How  should  these  factors  and  the  relationships  between  them  be  modeled? 

5.  Can  the  research  results  be  applied  US AF- wide? 

6.  How  can  the  now-quantified  extrinsic  factors  be  effectively  combined  with  the 
intrinsic  factors  to  develop  an  overall  asset  set  utility  or  value? 

Methodology 

This  research  involved  four  broad  phases  aimed  at  answering  the  six  research 
questions.  The  first  phase  was  comprised  of  a  form  of  Cognitive  Task  Analysis 
interviewing  known  as  Critical  Decision  Methodology  (CDM).  These  interviews  were 
conducted  with  34  campaign  planning  experts  from  the  Pentagon  (HQ  USAF),  four 
USAF  major  commands,  the  USAF  Air  Expeditionary  Force  Center  (AEFC),  and  three 
USAF  academic  “schoolhouses.”  The  purpose  of  the  interviews  was  three  fold:  to 
validate  the  extrinsic  factors  that  had  been  identified  in  previous  AFIT  research 
(discussed  in  Chapter  II),  to  identify  any  factors  that  had  not  been  previously  captured  by 
a  decision  support  model,  and  to  determine  appropriate  definitions  for  every  factor 
identified.  Post  interview  content  analysis  and  literature  review  were  the  primary  tools 
for  extracting  the  factors.  This  phase  addressed  research  questions  1,  2  and  5. 
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The  second  phase  of  research  consisted  of  a  literature  review  to  determine  the 
most  appropriate  Decision  Analysis  technique  with  which  to  model  this  decision 
situation.  This  phase  addressed  research  question  4. 

The  third  phase  of  research  was  conducted  by  in-person  interviews  and  e-mail 
questionnaires,  using  a  modified  “Delphi”  approach,  and  was  built  upon  the  foundational 
work  of  Phase  1 .  By  exploring  and  defining  the  relationships  between  the  factors, 
consistent  with  expert  responses  via  the  Delphi  questionnaires,  a  conceptual  map  or 
model  of  the  factors  was  developed.  The  completion  of  Phase  2,  due  in  large  part  to  the 
diversified  expert  pool,  addressed  research  question  3  and  laid  the  groundwork  for 
addressing  research  question  6. 

Phase  4  of  the  research  focused  on  final  development  of  the  decision  support 
model  and  explored  the  mathematical  relationships  between  the  proposed  decision 
support  model  and  the  intrinsic  value  of  a  candidate  force  mix  (research  question  6). 
Extensive  consultation  with  colleagues  performing  parallel  research  and  a  comprehensive 
literature  review  comprised  this  phase. 

Assumptions 

The  validity  of  this  decision  support  model  is  dependent  upon  the  assumption  that 
the  USAF  theme  of  Global  Engagement  does  not  dramatically  change.  That  is,  the  USAF 
currently  provides  for  combat  forces  to  respond  to  theater  crises  primarily  from  stations 
within  the  continental  United  States,  on  a  rotational  basis.  A  dramatic  change  would 
most  closely  approximate  a  return  to  the  pre-drawdown  military  doctrine  of  forward 
basing,  in  which  almost  all  required  active  duty  forces  are  already  in  place  in  the  theaters 
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of  crisis.  Such  a  scenario  could  lessen  or  eliminate  the  positive  impact  of  this  decision 
support  model.  However,  the  assumption  of  a  continuing  Global  Engagement  theme  is 
sound,  given  the  end  of  the  cold  war  and  the  relative  inefficiency  of  forward  basing. 

Scope  and  Limitations 

DARPA’s  ALP  initiative  encompasses  the  entire  Department  of  Defense.  The 
ALP  will  grant  streamlined,  automated  deployment  planning  capabilities  and  real-time 
asset  visibility  for  resources  spread  among  all  of  the  military  services.  However,  the 
efforts  of  this  research  are  limited  to  force  mix  selection  for  the  United  States  Air  Force 
only,  without  regard  to  other  services  or  other  service  capabilities.  On  a  highly  positive 
note,  this  research  applies  to  the  entire  Air  Force  rather  than  being  limited  to  a  single 
theater  of  operations  or  a  single  major  command,  as  was  the  case  with  previous  research 
in  this  area  (discussed  in  Chapter  II).  The  diversity  of  the  expert  pool,  as  well  as  the 
phrasing  of  interview  questions,  was  designed  to  provide  that  this  research  is 
representative  of  the  Air  Force  as  a  whole. 

Due  to  the  nature  of  this  research  and  the  fact  that  key  operating  instructions  and 
concepts  of  operations  are  classified,  highly  sensitive,  or  not  pertinent  to  this  study,  both 
the  research  conclusions  and  the  information  used  are  limited  to  an  unclassified  level. 

Although  the  focus  of  this  research  is  to  create  a  decision  support  model  that  will 
aid  campaign  planners  in  selecting  the  best  force  mixes,  the  scope  is  limited  to  only  those 
extrinsic  factors  that  are  important  to  such  decisions.  Intrinsic  factors  are  addressed  with 
respect  to  their  existence,  definition,  and  exclusivity  to  this  model,  but  are  not 
investigated  further.  Efforts  will  be  made  to  mathematically  relate  or  link  the  extrinsic 
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factors  to  potential  existing  intrinsic  values,  but  no  attention  will  be  given  to  the 
identification  or  development  of  an  intrinsically-oriented  decision  support  model. 

Finally,  the  basic  framework  for  USAF  expertise  within  this  research  is  limited  to 
current  doctrinal  and  instructional  USAF  and  Joint  publications,  and  USAF  campaign 
planning  experts  that  are  currently  active  in  their  respective  fields.  This  scope  limitation 
to  present-day  expertise  and  guidance  is  well-founded,  given  that  doctrine  and  instruction 
are  necessarily  gleaned  from  past  lessons.  Furthermore,  the  experts  presently  assigned  to 
planning  positions  or  doctrinal/planning  schoolhouses  are  the  very  individuals  who  are 
shaping  the  direction  of  contingency  planning  for  the  USAF  into  the  near-term  future. 

Summary 

This  chapter  provides  the  justification  and  motivation  for  developing  a  decision 
support  model  that  will  significantly  aid  campaign  planners  in  selecting  the  best  force 
mixes  for  contingencies.  The  critical  nature  of  campaign/deployment  planning  in  today’s 
Air  Force  is  undeniable.  This  research  quantifies  the  extrinsic  deployment  planning 
factors  and  integrates  them  into  an  executable  decision  model  that  will  simplify  and 
streamline  this  most  critical  process. 

Chapter  II  discusses  deliberate  and  crisis  action  planning  under  the  AEF  concept 
as  well  as  the  initial  extrinsic  and  intrinsic  factors,  then  provides  a  thorough  background 
discussion  of  DARPA,  ALP,  and  the  M-R  VAT.  The  chapter  also  covers  the  previous 
research  effort  that  generated  an  initial  collection  of  extrinsic  factors  and  an  executable 
decision  support  model.  The  chapter  ends  with  a  review  of  CDM  interviewing,  content 
analysis,  and  Delphi  studies — all  of  which  were  used  in  this  research. 
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Chapter  III  is  a  continuation  of  the  literature  review  that  focuses  on  the 
background  and  theory  of  decision  analysis,  and  reviews  a  number  of  candidate  decision 
analysis  tools  and  models,  providing  the  pros  and  cons  by  which  to  measure  each  tool’s 
suitability  to  this  research  problem.  The  chapter  ends  with  identification  of  the  selected 
decision  analysis  technique  for  this  research. 

Chapter  IV  lays  out  the  methodology  of  this  research.  It  begins  with  an  overview 
of  the  research  plan,  to  include  the  research  design,  and  then  covers  the  three  main  phases 
of  research  in  depth. 

Chapter  V  presents  the  results  of  each  research  phase.  It  goes  further  in 
describing  the  final  decision  support  model,  the  sensitive  analysis,  and  the  justification 
for  the  mathematical  concepts  used. 

Chapter  VI  provides  conclusions,  as  well  as  discussions  about  this  model’s 
applicability  to  DARPA,  ALP,  M-R  VAT,  and  the  USAF.  Recommendations  are  made 
as  to  how  to  integrate  this  model  with  existing  research  results  and  what  further  research 
is  most  appropriate. 
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II.  Literature  Review 


Introduction 

Campaign  planning  in  today’s  military  environment  is  both  challenging  and 
extraordinarily  critical,  with  the  decisions  of  planning  experts  not  only  affecting  the  lives 
and  well  being  of  friendly  forces  but  possibly  affecting  the  final  outcome  of  the  conflict 
itself.  In  the  USAF,  a  fundamental  mission  of  planners  at  all  echelons  is  to  provide  the 
Joint  Air  Component  Commander  and  or  Theater  Commander  a  set  of  combat  aircraft 
sufficiently  capable  of  meeting  theater  air  combat  requirements.  These  sets  of  combat 
aircraft  are  referred  to  throughout  this  work  as  force  mixes.  A  force  mix  is  defined  here 
as  any  combination  of  USAF  weapon  systems  that  collectively  possesses  the  inherent 
capability  to  meet  or  exceed  the  air  combat  requirements  for  a  specific  conflict  within  a 
specific  theater — as  determined  by  the  appropriate  campaign  planners.  This  definition  is 
important  when  noting  that  planners  must  select  force  mixes  that  are  inherently  capable 
of  accomplishing  a  given  mission.  The  central  theme  of  this  research  is  that  these 
planners  must  do  so  while  faced  with  extremely  short  time  constraints,  high  stress,  and  a 
large  number  of  extrinsic  factors  that  compound  an  already  complex  decision  situation. 

This  chapter  first  provides  an  overview  of  the  current  USAF 
deployment/contingency  planning  process  and  addresses  the  fundamental  problem:  the 
need  for  quick,  effective,  and  efficient  force  mix  selection.  It  also  defines  both  extrinsic 
and  intrinsic  factors,  and  provides  an  initial  listing  of  factors  for  each  type — in  their 
broadest  forms.  The  chapter  then  presents  an  overview  and  background  of  the  ALP  and 
the  M-R  VAT.  Together,  these  sections  serve  to  orient  the  reader  and  to  establish  the 
importance  and  support  for  this  research. 


10 


The  fourth  section  of  the  chapter  presents  a  background  of  a  previous  research 
effort  that  specifically  spurred  this  current  investigation.  The  chapter  concludes  by 
discussing  the  two  primary  knowledge  elicitation  techniques  used  during  this  study, 
Delphi  and  Critical  Decision  Methodology  (CDM). 

Deployment  Planning  Overview 

The  AEF  Perspective  on  Campaign  Planning.  Faced  with  a  continuous  threat 
from  the  Soviet  Union  and  Eastern  Bloc  during  the  Cold  War,  the  United  States  spent 
half  a  century  building  and  maintaining  a  significant  number  of  combat  resources  and 
military  personnel  overseas,  primarily  in  the  European  and  Pacific  theaters.  Such 
presence  was  deemed  necessary  to  counter  the  sizable  force-to-force  advantage  held  by 
communist  nations  in  these  respective  regions.  Today,  with  the  end  of  the  Cold  War, 
such  force-to-force  mismatches  no  longer  exist.  Although  legitimate  threats  to  the 
national  security  of  the  United  States  have  not  been  eliminated,  the  single,  most  central 
threat  has  been  reduced  sufficiently  to  generate  a  new  way  of  strategically  addressing 
national  defense. 

In  the  United  States  Air  Force  (USAF),  Global  Engagement  is  the  new  strategic 
direction.  It  is  aimed  at  supporting  the  country’s  national  interests  anywhere  in  the 
world,  any  time,  from  force  pools  located  primarily  within  the  continental  United  States 
(CONUS).  The  Air  Expeditionary  Force  (AEF)  concept  has  taken  hold  as  the  USAF’s 
primary  tool  in  carrying  out  the  potentially  formidable  tasks  presented  by  the  Global 
Engagement  theme — quick  regional  response  from  CONUS-based  forces,  regardless  of 
the  nature  and  extent  of  the  conflict.  Explicitly,  the  USAF  is  tasked  with  providing  air 
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and  space  forces  that  meet  specific  theater  requirements  across  the  full  spectrum  of 
military  possibilities  [Cohen,  1998]. 

Today,  the  AEF  concept  is  in  full  force,  with  the  establishment  of  the  AEF  Center 
(under  the  Air  Staff)  at  Langley  Air  Force  Base,  Virginia.  The  AEF  approach  cements 
the  USAF  change  in  strategy  from  forward  basing  with  abundant  resources  to  global 
engagement  with  scarcer  resources  and  greater  flexibility  [Looney,  1996:  5]. 

Within  just  48  hours  following  an  execute  order,  AEF  units  are  expected  to  have 
deployed  to  the  theater  and  commenced  combat  air  operations.  Typically,  an  AEF  unit’s 
force  mix  consists  of  36  aircraft  combining  Air  Superiority,  Air  Strike,  and  Suppression 
of  Enemy  Air  Defense  (SEAD)  mission  capabilities  [Godfrey,  1998:  1].  However,  such 
packages  can  be,  and  often  are,  tailored  to  meet  specific  requirements.  Regardless,  an 
AEF  unit  is  expected  to  conduct  either  defensive  or  offensive  air  operations 
independently  for  the  first  seven  days,  in  effect  buying  time  until  reinforcements  (if 
necessary)  become  available  [Godfrey,  1998:  2], 

It  is  easy  to  see  the  criticality  of  proper  force  mix  selection,  given  the  AEF 
environment.  The  AEF  must  respond  to  crises  ranging  across  the  full  spectrum  of 
potential  military  operations,  and  must  do  so  in  extremely  limited  time.  Compounding 
the  situation  is  the  fact  that  individual  deployable  AEF  force  mixes  may  soon  be  reduced 
to  six-,  four-,  or  two-ship  packages  rather  than  the  much  larger  Unit  Type  Codes  (UTCs) 
by  which  they  now  deploy  [Petersen,  2000].  UTCs  are  essentially  squadron-sized  (or 
slightly  smaller)  force  packages  that  planners  then  tailor  to  suit  their  individual 
requirements.  They  provide  for  resource  justification  and  accountability  and  are 
extremely  convenient  when  planners  are  pressed  for  time.  Given  the  current  initiatives  to 
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either  reduce  or  eliminate  UTCs  [Petersen,  2000],  timely  selection  of  the  best  force  mixes 
will  become  increasingly  challenging  and  potentially  overwhelming. 

Campaign  Planning — The  Force  Mix  Selection.  As  previously  mentioned,  force 
mix  selection  for  response  to  contingency  requirements  is  the  fundamental  problem 
addressed  by  this  research.  However,  force  mix  selection  is  just  part  of  a  complicated 
campaign  planning  process.  The  selection  of  a  force  mix  begins  a  chain  reaction  logistics 
response  aimed  at  getting  the  desired  forces  and  the  necessary  support  resources  to  the 
right  place  at  the  right  time.  Therefore,  force  mix  selection  might  have  the  single  greatest 
impact  on  the  size  of  the  deployment  footprint  and  the  deployment  speed,  as  well  as 
overall  force  sustainment  over  the  course  of  the  conflict.  Of  all  campaign  planning  tasks, 
force  mix  selection  is  of  prime  importance. 

Campaign  planners  at  all  levels  within  the  DoD  have  access  to  volumes  of  official 
publications  to  guide  the  planning  process.  The  joint  planning  environment  is  where  it  all 
begins  during  times  of  crisis.  Joint  operations  are  those  involving  cooperation  and 
interplay  within  theater  by  more  than  one  US  military  service.  The  joint  planning  process 
is  what  makes  this  possible,  and  it  serves  as  the  solid  bridge  along  the  command  trail  that 
extends  from  the  National  Command  Authority  (NCA),  through  the  Joint  Chiefs  of  Staff 
(JCS),  to  the  military  services  and  respective  Commanders  in  Chief  (CINCs)  of  given 
theaters  of  operation.  Joint  planning  is  a  systematic  procedure  used  by  relevant 
commanders  to  ascertain  the  best  solution  for  accomplishing  assigned  mission  tasks 
[JSOG,  1997:  para  500], 

The  Joint  Operation  Planning  and  Execution  System  (JOPES)  is  the  information, 
command,  and  control  tool  that  integrates  DoD  resource  and  readiness  information  to 
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meet  the  needs  of  joint  planning  decision  makers.  JOPES  expedites  the  development  of 
both  options  and  subsequent  operational  orders  during  crisis  action  planning,  and  is  the 
primary  management  tool  for  the  generation  and  execution  of  all  necessary  operations 
along  the  crisis  response  timeline,  including  mobilization,  deployment,  employment, 
sustainment,  and  redeployment  [JSOG,  1997:  para  506]. 

Planners  utilizing  JOPES  have  two  volumes  of  Joint  Pub  5-03.1,  Joint  Operation 
Planning  and  Execution  System,  that  describe  operation  of  the  system.  Volume  I 
provides  instruction  on  planning  policies  and  procedures.  Volume  II  deals  with  formats 
and  guidance.  Both  volumes  are  unclassified,  and  they  exhaustively  address — from  a 
generalized  joint  operations  perspective — the  full  spectrum  of  joint  planning  under 
JOPES. 

Official  guidance  for  addressing  both  the  planning  and  the  command  and  control 
issues  during  joint-level  crisis  action  planning  is  contained  in  Joint  Pub  3-56.1,  Command 
and  Control  of  Joint  Operations,  and  Joint  Pub  5-00,  Doctrine  for  Planning  Joint 
Operations.  Both  are  unclassified,  and  both  provide  broad  guidance  not  limited  to  JOPES 
usage. 

Clearly,  campaign  planners  have  ample  joint  operations  guidance.  However,  such 
guidance  is  not  specific  to  uniquely  USAF-oriented  operations.  Two  volumes  of  Air 
Force  Manual  10-401  serve  to  supplement  joint  instructions.  Volume  I,  Operation  Plan 
and  Concept  Plan  Development  and  Implementation,  provides  guidance  that  maintains 
the  necessary  joint  perspective  while  focusing  on  US  air  power  and  its  proper 
implementation.  Volume  II,  Planning  Formats  and  Guidance,  provides  instruction  on 
JOPES  utilization  to  generate  such  plans.  Further,  USAF  war  planners  draw  directly 
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from  the  basic  concepts  of  Air  Force  doctrine  in  matching  mission  requirements  to 
available  resources  in  times  of  crisis.  To  this  end,  all  USAF  planners  keep  and  maintain 
copies  of  Air  Force  Doctrine  Document  1,  Basic  Air  Force  Doctrine  [AFDD  1, 1997], 
and  Air  Force  Doctrine  Document  2  [AFDD  2,  1998],  Organization  and  Employment  of 
Aerospace  Power. 

Although  comprehensive  and  thoroughly  representative,  this  listing  of 
publications  available  to  campaign  planning  experts  is  by  no  means  exhaustive.  It  is 
merely  a  snap  shot  of  the  wealth  of  information  and  tools  available  to  decision  makers 
today,  and  it  serves  to  illustrate  the  potential  complexity  of  campaign  planning  and  force 
mix  selection.  This  research  depended  heavily  on  content  analysis  of  many  sections  of 
these  publications,  as  well  as  an  overview  reading  of  each.  However,  aside  from  the 
identification  of  extrinsic  factors  and  intrinsic  factor  categories  to  be  addressed  later, 
these  publications  will  not  be  discussed  in  depth. 

Although  force  mix  selection  is  just  one  part  of  a  complicated  planning  process 
that  includes  use  of  JOPES  and  many  official  publications  as  guidance,  it  is  indeed  this 
selection  of  forces  that  drives  the  requirements  for  the  entire  deployment  and 
employment  process.  Decision  makers  need  a  tool,  a  decision  model,  that  incorporates 
both  the  judgement  of  seasoned  planning  experts  and  the  guidance  of  official  publications 
in  order  to  very  quickly  and  effectively  select  the  best  force  mixes  to  meet  requirements. 

Planning  Decision  Factors:  Extrinsic  Versus  Intrinsic.  It  is  proposed  that  the 
factors  to  be  evaluated  in  determining  the  best  force  mix  can  be  divided  into  two 
categories,  intrinsic  and  extrinsic.  Because  this  research  is  limited  to  the  creation  of  a 
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decision  model  for  extrinsic  factors  only,  it  is  necessary  to  clearly  define  both  types  of 
factors. 

Definition  1 :  Intrinsic  factors  represent  a  weapons  platform’s  fundamental  ability 
to  accomplish  a  specified  aerospace  mission  or  tasking  [Buzo,  2000:  2],  and  are  limited  to 
determinations  of  efficacy  in  a  given  situation  related  only  to  the  designated  purpose  of 
its  design.  For  example,  an  A-10  aircraft’s  fundamental  ability  is  Air-to-Ground  Attack; 
its  intrinsic  value  for  such  missions  is  high  because  Air-to-Ground  Attack  is  the 
designated  purpose  of  an  A-lO’s  design. 

Definition  2:  Extrinsic  factors  are  situational  considerations,  external  to  and 
independent  of  the  purpose  of  a  weapons  platform’s  design,  that  affect  determination  of 
the  appropriateness  and  subsequent  goodness  of  that  weapons  platform  for  a  specified 
mission — given  a  specific  contextual  environment  that  requires  evaluation  [Buzo,  2000: 
2-3].  For  example,  an  A-10  aircraft’s  extrinsic  value,  given  a  situation  where  the 
operating  location  is  2,000  miles  from  the  primary  targets  and  30mm  ammunition  is 
unavailable,  might  be  quite  low  due  to  the  A-lO’s  lack  of  speed  and  need  for  30mm 
ammunition. 

Definitions  1  and  2  were  refined  via  interviews  with  subject  matter  experts  during 
the  early  phases  of  research.  Although  the  final  forms  are  presented  here,  discussion  of 
their  development  is  reserved  for  Chapter  IV. 

Intrinsic  and  extrinsic  factors  form  a  natural,  logical  separation  during  the  course 
of  crisis  action  planning  (CAP).  Given  an  overarching  mission  in  response  to  a  crisis, 
planners  immediately  conceptualize  potential  force  mixes  based  upon  factors  related  only 
to  the  individual  weapons  platforms’  capabilities  to  accomplish  the  tasks  called  for,  a  sort 
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of  “pie  in  the  sky”  wish  list  of  combat  forces  [Duvall,  2000;  12  AF,  2000].  This  early 
process  is  more  “artwork  than  science,”  and  leads  to  course  of  action  (COA)  development 
based  initially  on  the  force  mixes  desired  [Duvall,  2000].  This  subsequent  COA  then 
introduces  situational  factors  and  logistical  requirements — the  extrinsic  factors — in 
accomplishment  of  various  feasibility  analyses  [12  AF,  2000]. 

Consider  a  campaign  planner’s  need  to  provide  a  force  mix  capable  of 
accomplishing  air  interdiction.  Candidate  weapons  platforms  can  be  considered  solely  on 
the  basis  of  their  designed  capability  to  accomplish  this  aerospace  mission.  The  F-15E 
Strike  Eagle  is  certainly  qualified  for  air  interdiction  when  considered  independently  of 
any  other  situational  factors.  The  A-10A  ground  attack  aircraft  is  not.  The  platform’s  . 
designed  capability  is  thus  an  intrinsic  factor. 

In  consideration  of  time  limitations  and  resource  constraints,  this  research  is 
limited  to  extrinsic  factors  only.  However,  the  question  of  intrinsic  value  is  being 
investigated  by  others  at  AFIT  in  support  of  the  ALP,  concurrently  with  this  research. 

Given  a  weapons  platform  that  is  intrinsically  capable  of  accomplishing  a  desired 
aerospace  mission,  the  extrinsic  factors  affecting  the  situation  could  determine  the  overall 
suitability  of  the  platform.  For  example,  an  excessive  distance  of  the  staging  base  from 
the  desired  targets  may  negate  the  use  of  the  desired  platform,  even  though  the  platform 
is  intrinsically  capable.  If  the  staging  base  in  question  is  the  only  available  operating 
location  option,  then  this  extrinsic  factor  would  affect  the  selection  of  the  force  mix. 

This  research  posits  that  extrinsic  factors  can  be  quantified  and  valued 
independently  of  intrinsic  factors  because  all  extrinsic  factors  are  situational,  and  any 
candidate  force  mix  can  be  weighed  against  a  given  situation.  However,  the  relationship 
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between  a  situation  and  its  impact  on  a  candidate  force  mix  requires  study.  One  can  be 
evaluated  without  the  other,  but  final  selection  of  a  best  force  mix  cannot  occur  without  a 
final  integration  of  the  two.  Hence,  force  mixes  will  have  separate  intrinsic  and  extrinsic 
utilities  that  will  induce  potential  tradeoffs  in  value  that  result  in  an  overall  value  for  a 
given  force  mix  in  a  given  situation. 

As  the  central  starting  point  for  determining  the  decision  factors  which  form  the 
foundation  of  this  research,  content  analysis  was  accomplished  on  Air  Force  Doctrine 
Document  2  [AFDD  2, 1998:  76],  Air  Force  Doctrine  Document  1  [AFDD  1,  1997:  46- 
60],  Joint  Operation  Planning  and  Execution  System  [JPUB  5-03.1, 1994:  P-1  to  P-5], 
and  A  Decision  Support  Tool  to  Aid  Campaign  Planners  in  Selecting  Combat  Aircraft  for 
Theater  Crisis  [Buzo,  2000:  42],  The  content  analysis  yielded  sets  of  intrinsic  and 
extrinsic  decision  factors  that  campaign  planners  must  consider  in  times  of  crisis  (see 
Figure  1) 

The  factors  gleaned  are  broad  and  general,  more  probably  serving  as  categories  of 
decision  factors  rather  than  factors  in  and  of  themselves.  Such  broadness  is  desirable  at 
this  point  because  it  presents  a  fundamental  starting  point.  Notice  upon  examination  of 
Figure  1  that  both  extrinsic  and  intrinsic  factors  or  categories  are  identified.  This 
research  focuses,  from  this  point  forward,  on  only  the  extrinsic  factors.  Perhaps  a 
number  of  future  studies  will  be  necessary  in  order  to  properly  address  the  intrinsic 
decision  factors  of  campaign  planning.  Figure  1  presents  the  initial  factors. 
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Figure  1 :  Extrinsic  and  intrinsic  decision  factors/issues 


The  Advanced  Logistics  Project  and  the  M-R  VAT 

The  general  factors  identified  in  Figure  1  provide  a  starting  point  for  discussing 
force  mix  selection  criteria.  Crisis  action  planning  for  contingency  deployments,  for  both 
joint  operations  and  within  the  Air  Force,  is  a  complicated  task  involving  a  myriad  of 
complex  decisions  under  high  pressure  situations,  often  with  severe  time  constraints.  The 
selection  of  force  mixes  to  best  accomplish  the  mission  is  the  prime  prerequisite  for 
development  of  the  overall  logistics  plan.  Errors  or  miscalculations  in  the  initial  selection 
of  force  mixes  may  create  undesirable  repercussions  in  the  logistics  plan  development  to 
follow,  lending  criticality  to  the  importance  of  force  mix  selection  decision  support  tools. 

The  Defense  Advanced  Research  Projects  Agency  (DARPA)  is  in  its  fifth  year  of 
developing  the  Advanced  Logistics  Project  (ALP),  a  technological  approach  to 
significantly  reducing  the  logistics  planning  process  time  that  could  change  the  face  of 
deployment  planning  for  all  military  departments  [Carrico,  1999].  The  ALP  will  be  a 
“distributed  computing  architecture  that  will  create  a  near-real-time  deployment  planning 
process  for  military  forces  by  enabling  logistics  planners  from  the  US  military  services  to 
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quickly  and  efficiently  develop  and  implement  a  situation-tailored  logistics  plan” 
[Carrico,  1999:  1]. 

The  ALP  is  an  automated  system  that  will  integrate  all  logistics  missions  and 
organizations,  utilizing  a  single  system  to  expedite  seamless  planning  and  execution 
[Shaneman,  1999:  6].  An  “object-oriented  design  methodology”  uses  simplified  but 
well-defined  components  to  spur  comprehensive  logistics  management  across  the  full 
system.  These  components  are  the  “basic  building  blocks,”  known  as  the  ALP  clusters, 
that  represent  combat  units,  support  units,  and  command  and  control  responsibilities 
[Shaneman,  1999:  6].  For  the  USAF,  the  ALP  contingency  process  begins  when  the 
deployment  order  is  initiated.  Clusters  for  deploying  units,  in  real  time,  fill  the  required 
AEF  clusters  with  forces  and  resources.  The  AEF  cluster  immediately  tailors  all 
requirements  based  on  decision  rules  and  sends  these  requirements  to  the  Logistics 
Readiness  Center  (LRC)  cluster.  The  LRC  cluster  sources  all  requirements  using 
optimization  techniques  and  submits  the  results  to  the  US  Transportation  Command 
(TRANSCOM)  cluster.  Finally,  the  TRANSCOM  cluster  integrates  these  requirements 
with  those  of  the  other  services’  deploying  forces,  optimizing  the  sequence  and  speed  of 
airlift  while  reducing  the  deployment  footprint  [Shaneman,  1999:  7]. 

The  ALP  presented  a  “landmark”  demonstration  in  1998,  building  a  Level  5  (the 
most  detailed)  logistics  plan  for  the  US  Army’s  3rd  Infantry  Division  from  their  home 
station  through  full  administrative  load  onto  a  ship  in  port.  Using  a  small  set  of 
directives,  ALP  produced  the  required  high-detail  logistics  plan  in  less  than  one  hour 
[Carrico,  1998:  5]. 
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It  is  important  to  note  that  the  demonstration  used  standard  internet  bandwidth 
and  medium  capability  personal  computers,  and  built  the  detailed  plan  based  on  live  data 
from  the  Joint  Total  Asset  Visibility  (JTAV),  Global  Transportation  Network  (GTN),  and 
Global  Decision  Support  System  (GDSS)  databases  [Carrico,  1998:  5].  The  ALP  will 
quickly  provide  joint  detailed  logistics  plans,  the  results  of  which  can  then  be  transmitted 
instantaneously  to  all  of  the  critical  players  to  generate  significantly  reduced  deployment 
timelines. 

However,  the  process  as  it  stands  now  begins  with  the  deployment  order.  The 
ALP  has  not  yet  attained  the  ability  to  expedite  selection  of  the  best  logistics  plan  from 
among  competing  alternatives  [Buzo,  2000: 1].  Selection  of  the  best  logistics  plan  cannot 
occur  without  beginning  the  process  with  selection  of  the  best  force  mix. 

Recognizing  the  logical  requirement  of  having  a  best  force  mix  as  input  into  the 
ALP  architecture,  DARPA  enlisted  the  aid  of  the  Air  Force  Institute  of  Technology 
(AFIT).  Lieutenant  Colonel  Alan  Johnson  and  Major  Stephen  Swartz,  both  assistant 
professors  at  AFIT,  seized  this  tasking  in  conceptualizing  the  Mission-Resource  Value 
Assessment  Tool  (M-R  VAT).  The  M-R  VAT  will  assign  relative  values  to  critical 
resources  over  time,  better  coordinating  the  time-phased  arrivals  of  resources  on  given 
days  of  the  contingency  with  actual  requirements  for  those  given  days  [Swartz,  2000:  1]. 

The  most  fundamental  concept  embraced  by  the  M-R  VAT  is  the  time  value  of 
logistics  [Swartz,  2000:  1].  That  is,  the  value  of  a  resource  changes  over  time  as  a 
campaign  progresses.  Because  the  nature  of  the  mission  changes  over  the  course  of  a 
conflict,  the  value  of  required  combat  resources  must  change  to  reflect  this  [Swartz, 
1999b:  1].  Existing  time  phased  force  deployment  documents  (TPFDDs)  already 
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prioritize  resource  movement  to  theaters,  but  such  prioritization  is  based  on  the  criticality 
of  combat  units  rather  than  the  time-phased  criticality  of  resources  within  these  units. 

The  M-R  VAT  shifts  this  prioritization  paradigm.  It  will  go  further  by  decreasing  the 
logistics  footprint  in  the  crucial  early  days  of  airlift  operations  by  enabling  commanders 
to  select  the  force  mixes  that  possess  the  best  time-phased  utility  before  they  request 
resource  allocations. 

Initialization,  Analysis,  and  Monitoring/Replanning  comprise  the  three  phases  of 
the  M-R  VAT  operation,  explained  by  Major  Swartz: 

During  the  Initialization  Phase,  the  M-R  VAT  elicits  critical  information 
from  the  user  and  structures  the  nature  of  the  specific  situation  to  be  addressed. 
The  Analysis  Phase  consists  of  determining  the  best  set  of  solutions  for  the 
problem,  and  assisting  the  decision  maker  in  considering  tradeoffs  before  making 
the  final  selections.  During  Monitoring  and  Replanning,  the  system  databases  are 
updated  as  the  plan  is  executed.  The  decision  maker  is  presented  with  a  running 
summary  of  plan  execution,  and  he  or  she  is  able  to  re-solve  the  problem  as  the 
situation  changes.  [Swartz,  2000:  2] 

This  research  is  in  direct  support  of  the  M-R  VAT.  Specifically,  the  extrinsically- 
centered  decision  support  tool  developed  here  will  provide  the  decision  factors  to  be  used 
by  the  campaign  planner  in  addressing  the  specific  situation  during  the  M-R  VAT 
Initialization  Phase.  Furthermore,  the  decision  support  tool  will  provide  a  scored  set  of 
feasible  solutions  during  the  M-R  VAT  Analysis  Phase. 


The  Existing  Decision  Support  Tool  for  M-R  VAT 

This  research,  aimed  at  developing  an  extrinsically-oriented  decision  support 
model,  is  a  direct  follow-on  to  Lieutenant  Christopher  Buzo’s  work,  A  Decision  Support 
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Tool  to  Aid  Campaign  Planners  in  Selecting  Combat  Aircraft  for  Theater  Crisis  [Buzo, 

2000], 

In  his  work,  Buzo  used  content  analysis  of  JPUB  5-03.1  and  AFDD  2  to  establish 
the  minimum,  basic  set  of  extrinsic  considerations  necessary  for  campaign  planning 
[Buzo,  2000:  42].  He  then  used  modified  Cognitive  Decision  Method  (CDM) 
interviewing  techniques  to  elicit  additional  extrinsic  factors  related  to  campaign  planning 
from  25  USAF  subject  matter  experts  [Buzo,  2000:  16-20, 46].  The  interviews  also 
served  to  refine,  define,  and  expand  those  factors  that  had  been  previously  identified 
through  content  analysis.  All  of  the  subject  matter  experts  were  USAF  officers  holding 
the  ranks  of  Major  through  Colonel  (0-4  through  0-6),  and  were  drawn  from  Air  Combat 
Command,  Air  Force  Central  Command,  Air  University,  and  Air  Staff  [Buzo,  2000:  46]. 

Challenged  with  developing  an  executable  decision  support  tool  using  these 
extrinsic  factors,  Buzo  investigated  two  Decision  Analysis  (DA)  models  [Buzo,  2000]. 
His  candidate  models  included  Thomas  L.  Saaty’s  Analytic  Hierarchy  Process  [Saaty, 
1982],  and  a  form  of  Multi -Attribute  Utility  Theory  (MAUT)  that  utilizes  Value  Focused 
Thinking,  developed  by  Ralph  Keeney  [Keeney,  1992], 

Citing  several  potential  shortcomings  with  the  Analytic  Hierarchy  Process  (AHP), 
Buzo  selected  Keeney’s  Value  Focused  Thinking  (VFT)  application  of  MAUT  [Buzo, 
2000:  20-21].  Lieutenant  Colonel  Jack  Kloeber,  Associate  Professor  of  Operations 
Research  at  AFIT,  advocated  the  use  of  VFT  over  other  Decision  Analysis  methods  and 
helped  to  develop  the  theoretical  foundations  of  Buzo’s  work.  VFT’s  appeal  is  that  it 
provides  a  method  for  quantifying  otherwise  qualified  objectives,  rather  than  alternatives, 
in  structuring  a  decision  maker’s  values — a  hierarchical,  objectives-oriented  approach  to 
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decision  making  that  integrates  all  of  the  important  considerations  [Keeney,  1992;  Buzo, 
2000:  21].  Both  AHP  and  the  VFT  application  of  MAUT  are  discussed  as  decision 
model  alternatives  in  Chapter  III. 

Having  used  VFT  techniques  to  establish  a  hierarchy  of  factors  representing  the 
extrinsic  decision  objectives  of  force  mix  selection,  Buzo  conducted  additional  interviews 
with  the  subject  matter  experts  to  obtain  scores,  weights,  and  relationships  for  and  among 
the  factors  [Buzo,  2000:  44].  An  important  consideration  is  that  Buzo  concentrated  these 
follow-up  VFT  model-building  efforts  on  the  expertise  of  USAF  Central  Command 
planners,  intentionally  limiting  the  scope  and  applicability  of  his  decision  support  tool  to 
theater  operations  in  Southwest  Asia  [Buzo,  2000:  6, 44]. 

Crisis  Action  Planning  for  contingency  deployments  in  the  USAF  involves  group 
decision  making.  Furthermore,  the  decentralization  of  responsibilities  and  authority 
among  the  USAF  Major  Commands  may  have  resulted  in  individual  sets  of  business 
rules  that  are  not  universally  applied  throughout  the  USAF.  For  these  reasons,  Buzo  used 
a  modified  Delphi  study  (comprised  of  3  rounds  of  questionnaires)  that  generated  subject 
matter  expert  agreement  on  the  structure,  scores,  and  weights  used  in  his  VFT  decision 
support  tool  [Buzo,  2000:  44,  55-63],  However,  only  ten  of  the  original  25  subject  matter 
experts  participated  through  the  final  round  of  questionnaires  [Buzo,  2000:  81]. 

Buzo’s  extensive  use  of  Craig  W.  Kirkwood’s  Strategic  Decision  Making: 
Multiobjective  Decision  Analysis  with  Spreadsheets  [Kirkwood,  1997]  enabled  him  to 
create  a  spreadsheet-based,  executable  decision  support  tool  that  quantified  and  related 
the  extrinsic  decision  factors  that  are  important  to  campaign  planning.  Such  a  tool  holds 
extreme  value  in  today’s  military  operations  planning  environment  and  could  be 
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integrated  into  the  M-R  VAT  and  subsequently  linked  to  the  ALP.  Buzo’s  work  puts  the 
USAF  one  step  closer  to  optimal  force  mix  selection  and  near-instantaneous  logistics  plan 
development  and  execution  through  ALP  and  the  M-R  VAT.  This  one  step  sets  the  stage 
for  further  research. 

Given  a  wide  range  of  DA  tools,  techniques,  and  models  available,  Buzo 
examined  the  merits  of  only  two  [Buzo,  2000:20-21].  This  current  research  improves 
upon  that  effort  by  investigating  a  number  of  candidate  DA  tools,  beginning  with  an 
overview  of  the  field  and  a  general  categorization  of  techniques. 

Additionally,  Buzo  created  a  single  VFT  hierarchy  using  over  20  extrinsic 
decision  factors  gleaned  from  his  research  [Buzo,  2000:  80,  105-1 12].  Although  VFT 
may  be  a  desired  DA  methodology  in  this  arena,  the  initial  research  did  not  investigate 
the  possibility  that  some  factors  may  not  belong  within  the  hierarchy,  or  may  at  least  be 
better  suited  to  inclusion  within  a  co-hierarchy  or  subset  that  integrates  with  the  main 
hierarchy.  Some  of  the  factors  may  more  appropriately  serve  as  a  filter  or  screen  for  the 
main  tool.  The  existence  and  importance  of  the  factors  notwithstanding,  it  is  their  place 
and  function  within  the  decision  support  tool  that  may  be  in  question.  It  is  important  to 
inteiject  here  that  the  extrinsic  categories  identified  in  Figure  1,  gleaned  from  content 
analysis,  are  the  exact  starting  categories  that  had  been  identified  by  Buzo. 

The  mathematical  relationships  that  drive  the  Buzo  decision  support  tool  are 
drawn  almost  exclusively  from  Kirkwood’s  instruction  and  represent  modified  MAUT 
[Kirkwood,  1997].  The  factors  and  sub-factors  are  additive.  This  approach  is  not 
inherently  flawed.  However,  the  current  research  effort  addresses  other  potential 
mathematical  relationships,  between  and  among  factors  and  sub- factors,  as  they  might 
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apply  to  this  specific  context.  The  goal  is  not  to  introduce  altogether  different  equations 
and  formulas,  rather  the  goal  is  to  avoid  exclusion  of  other  mathematical  functions  that 
might  enhance  model  performance. 

Another  area  of  potential  improvement  is  the  subject  matter  expert  pool  itself. 
The  Buzo  model  was  built  upon  the  expertise  of  representatives  from  just  two  major 
commands,  one  schoolhouse,  and  the  Air  Staff.  Furthermore,  the  actual  scores  and 
weights  of  factors  were  decided  by  experts  from  just  the  Air  Force  Central  Command 
[Buzo,  2000:  44].  This  was  the  intended  scope  of  Buzo’s  work,  and  he  does  not  claim 
universal  applicability  to  the  entire  USAF.  However,  the  goal  of  this  current  research  is 
just  that — universal  application  for  the  USAF.  To  that  end,  the  subject  matter  pool  is 
larger  and  represents  the  four  operational  major  commands  (within  CONUS),  two 
schoolhouses,  and  the  Air  Staff. 

This  literature  review  has  provided  an  overview  of  crisis  action  campaign 
planning  and  illustrated  its  criticality  to  USAF  operations.  It  has  also  given  an  overview 
of  the  ALP  and  the  M-R  VAT,  the  driving  forces  behind  this  research.  The  preceding 
section  discussed  the  original  extrinsically-oriented  decision  support  tool  that  was 
developed  to  meet  the  needs  of  M-R  VAT  and  ALP.  Next  is  a  discussion  of  the 
knowledge  elicitation  techniques  used  for  this  current  research. 

Eliciting  Knowledge:  Interviewing/ CDM,  Content  Analysis,  and  Delphi 

Interviewing/CDM  and  Content  Analysis.  The  Critical  Decision  Method  (CDM) 
of  semi-structured  interviewing  was  developed  in  1989  by  Klein,  Calderwood,  and 
MacGregor.  It  is  an  expert  decision  and  judgement  centered  approach  to  the  Cognitive 
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Task  Analysis  (CTA)  methodology  [Buzo,  2000:  16],  a  methodology  that  enhances  the 
study  of  non-observable  skills  and  expert  thinking  [Seamster,  1997:  25]. 

CDM  goes  beyond  traditional  CTA  in  that  it  is  a  retrospective  and  introspective 
interviewing  approach  that  applies  cognitive  probes  to  the  critical  decision  points  that 
require  expert-level  decisions  or  judgements  [Klein  et  al,  1994:  464]. 

The  processes  of  making  decisions  and  solving  problems  on  the  level  of  force  mix 
selection  are  complex  and  highly  cognitive  tasks  that  require  considerable  skills.  They 
are  the  processes  that  actually  drive  the  motor  behaviors  that  have  been  the  subject  of 
traditional  task  analyses  in  the  past  [Seamster,  1997:  78].  The  important  issues  to  this 
research  are  the  previous  experiences  of  subject  matter  experts  and  their  expert  responses 
to  (and  their  lessons  learned  from)  critical  decision  making  opportunities,  not  just  the 
motor  tasks  or  structurally  imposed  behaviors.  Hence,  CDM,  with  its  emphasis  on 
retrospective  cognitive  responses,  is  ideally  suited  to  this  research. 

In  conducting  CDM  interviewing,  the  interviewer  is  tasked  with  eliciting 
comprehensive  job  information  from  the  respondent.  This  is  best  accomplished  in  a 
semi-structured  environment  in  which  the  interviewer  uses  his  or  her  specific  knowledge 
of  the  information  needed  to  guide  the  respondent  to  appropriate  personal  reference 
points.  The  respondent  is  asked  to  relay  his  or  her  personal  experience,  given  the  now 
developed  reference  point  that  involves  critical  decision  making  opportunities. 

The  interviewer  uses  the  situational  reference  point  to  probe  decision  making 
opportunities,  specific  to  the  subject  matter  expert’s  personal  experience.  The  challenge 
is  to  remain  on  subject,  with  the  interviewer  getting  the  respondent  to  answer  the 
questions  posed  [Interview  Research  Manual,  1976:  15].  In  dealing  with  incomplete  or 
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inappropriate  responses,  the  interviewer  repeats  the  question,  exhibits  expectant  pause, 
repeats  the  respondent’s  reply,  asks  a  neutral  question,  or  asks  outright  for  clarification 
[Interview  Research  Manual,  1976:  15-16], 

Questions  might  arise  as  to  the  reliability  and  validity  of  information  gleaned 
from  CDM  interviewing  to  obtain  extrinsic  decision  factors  for  crisis  action  campaign 
planning.  However,  some  experts  clearly  advocate  the  process.  According  to  Canter, 
Brown,  and  Groat,  “Research  would  be  more  effective  if  procedures  allowed  the 
interviewees  to  express  their  own  view  of  the  issues  at  hand,  in  their  own  way,  whilst  still 
providing  information  that  is  structured  enough  for  systematic  analysis  and  reporting” 
[Canter  et  al,  1985:  83]. 

The  combination  of  CDM  interviewing  and  content  analysis,  when  done  properly, 
provide  for  relatively  sound  reliability  and  validity.  According  to  Brenner,  Brown,  and 
Canter,  the  processes  of  content  analysis  and  multidimensional  scaling  (done  in  this 
research  via  decision  analysis)  make  it  possible  to  develop  systematic  and  quantitative 
summaries  of  data  that  “would  not  have  been  amenable  to  analysis  in  the  past,  when  such 
data  might  have  been  dismissed  as  ‘too  qualitative.’  These  data  can  now  form  the  basis 
of  research  activity”  [Brenner  et  al,  1985:  1].  Open-ended,  largely  unstructured  research 
material  should  not  impose  “unnatural  restraints”  on  a  researcher’s  analysis  [Mostyn, 
1985:  115].  The  researcher’s  purpose  is  to  systematically  identify  the  specific  elements 
of  communication  so  that  conversion  into  scientific  data  can  be  objectively  accomplished 
[Mostyn,  1985:  117]. 

This  research  probes  the  experiences  of  subject  matter  experts  using  CDM 
interviewing  and  content  analysis  in  order  to  develop  an  exhaustive  listing  of  extrinsic 
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decision  factors  necessary  to  consider  during  the  force  mix  selection  portion  of  crisis 
action  planning.  Buzo’s  work  on  the  initial  decision  support  tool  followed  the  same  early 
process  [Buzo,  2000],  Elicitation  using  CDM  interviewing  is  a  sound  approach  to 
gathering  the  extrinsic  factors  necessary  to  build  a  decision  support  tool.  Content 
analysis  of  the  interviews,  as  well  as  the  official  guidance  provided  for  deployment 
planning,  is  the  intermediate  step  in  gathering  and  quantifying  the  factors.  Final 
quantification  of  factors  involves  the  application  of  DA  techniques,  discussed  at  length  in 
Chapters  III  and  IV. 

The  Delphi  Study.  Modified  Delphi  techniques  comprised  the  preferred  method 
for  gaining  a  consensus  of  this  current  decision  support  tool  from  among  the  selected 
campaign  planning  experts.  Linstone  and  Turoff  define  the  Delphi  technique  as 
“structured  communication  that  allows  a  group  of  individuals,  as  a  whole,  deal  with  a 
complex  problem”  [Linstone  and  Turoff,  1975:  3].  Norman  Dalkey,  one  of  the 
originators  of  the  Delphi  method,  states  that  the  rationale  for  the  use  of  the  technique  is  to 
obtain  the  intuitive  insights  of  experts  and  to  use  their  judgement  as  systematically  as  one 
can  [Adler,  1996:  4].  In  order  to  pool  the  opinions  and  expertise  of  individuals  within  a 
group  into  a  central  theme  or  direction,  Delphi  techniques  allow  feedback  of  individual 
contributions  of  knowledge,  provide  assessment  of  the  group  judgement  or  view,  allow 
individuals  to  then  revise  their  views  in  light  of  other  views  or  a  developing  central 
theme,  and  provide  for  anonymity  among  individual  participants  [Linstone  and  Turoff, 
1975:  3], 

What  the  Delphi  technique  offers  is  what  is  specifically  required  in  this  research 
situation:  access  to  the  opinions  and  judgements  of  subject  matter  experts,  controlled 
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feedback  to  refine  these  judgements,  and  a  necessary  measure  of  anonymity  to  assure 
honest  responses  that  are  free  from  the  pressures  to  conform  to  group  dynamics.  The 
latter  point  is  particularly  important  in  a  military  setting.  In  group  dynamics,  a  dominant 
individual  can  often  influence  the  opinions  of  others  [Brown,  1968:  2-3].  The  effects  of 
individual  dominance  can  be  amplified  by  the  clear  rank  structure  within  the  military, 
leading  subordinates  to  outwardly  concur  with  the  opinions  of  their  supervisors  even 
though  they  may  privately  disagree.  Delphi  techniques  eliminate  this  effect  because  all 
responses  are  anonymous  [Brown,  1968:  3]. 

Linstone  and  Turoff  provide  several  reasons  for  using  the  Delphi  method.  The 
first  is  when  the  problem  cannot  be  addressed  with  “precise  analytical  techniques,”  but  is 
better  suited  to  the  collection  of  individual  subjective  judgements  [Linstone  and  Turoff, 
1975:  4].  Given  the  need  for  wide-ranging  expertise  on  decision  making  during 
campaign  planning,  this  reason  certainly  applies  to  this  research.  The  second  reason 
offered  is  when  more  experts  are  needed  than  can  interact  in  a  committee-type  exchange 
or  face-to-face  group  interview  [Linstone  and  Turoff,  1975:  4].  Again,  this  applies  to  this 
research  because  campaign  planning  experts  are  currently  on  duty  in  their  respective 
critical  fields,  and  spread  among  a  number  of  offices  and  agencies  across  the  United 
States.  The  third  reason,  similar  to  the  second  reason  in  substance  and  applicability  here, 
is  when  the  time  and  cost  are  sufficiently  high  to  preclude  frequent  group  meetings 
[Linstone  and  Turoff,  1975:  4],  The  final  reason  to  use  the  Delphi  method  is  to  avoid 
group  domination  by  quantity  (majority),  personality,  or  position  [Linstone  and  Turoff, 
1975:  4].  Again,  given  the  clear  rank  structure  and  disciplined  superior-subordinate 
relationships  in  the  military,  this  fourth  reason  is  especially  applicable  to  this  research. 
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Having  established  the  suitability  of  the  Delphi  method  for  this  research,  it  is 
necessary  to  discuss  the  process. 

Generally  put,  the  Delphi  method  is  used  to  collect  and  distill  the  knowledge  of 
subject  matter  experts  by  using  controlled  opinion  feedback  that  is  interspersed  within  a 
series  of  questionnaires  [Adler,  1996:  3].  The  Delphi  technique  is  usually  executed  in  3-4 
rounds  that  may  or  may  not  involve  the  two  general  phases  of  exploration  and  evaluation 
[Turoff,  1975:  89].  Given  a  generalized  problem  with  no  central  theories  in  which 
neither  the  issues  nor  the  solutions  are  clear,  an  exploration  phase  can  approximate  the 
start  of  ungrounded  research.  However,  a  study  in  which  theory  discovery  is  not  a  goal 
and  in  which  issues  are  clearly  identified  need  only  use  an  evaluation  phase,  bypassing  an 
unnecessary  exploration. 

The  first  round  of  Delphi  is  “seeded”  with  an  initial  range  of  options  (in  general, 
but  particularly  in  this  research),  but  allows  individuals  almost  absolute  freedom  to 
expand  or  alter  the  list  [Turoff,  1975:  89].  During  this  round,  group  members  come  to 
understand  the  goals  of  the  study,  as  well  as  the  terminology  and  taxonomy,  and  they 
examine  the  subject  issues  to  validate  their  personal  competency  levels  with  regard  to  the 
subject  and  direction  of  the  study. 

The  responses  of  the  experts  to  the  Delphi’s  first  round  begin  the  controlled 
feedback  process  that  is  an  important  aspect  of  this  technique.  The  controlled  feedback 
process  reduces  extraneous  or  redundant  material  that  might  otherwise  obscure  the  more 
significant  material  offered  by  the  experts  [Dalkey,  1967:  3],  The  researcher  filters  the 
extraneous  or  irrelevant  information  and  summarizes  the  important  contributions,  re¬ 
submitting  the  results  to  the  expert  group.  This  second  round  provides  experts  with 
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points  of  view  they  may  not  have  considered,  while  also  providing  all  of  the  group 
experts  a  compilation  of  ideas  that  may  begin  to  resemble  either  a  central  them  or  a 
divergence  in  schools  of  thought  [Dalkey,  1968:  4], 

How  does  this  work?  Each  member  must  justify  their  responses  (anonymously), 
which  are  then  made  available  to  all  other  group  members.  Through  Round  2  and 
subsequent  iterations  of  the  process,  experts  begin  to  take  into  account  “considerations 
they  might  through  inadvertence  have  neglected,  and  to  give  due  weight  to  factors  they 
were  inclined  to  dismiss  as  unimportant  at  first  thought”  [Brown,  1968:  2-7]. 

The  number  of  rounds  used  to  accomplish  a  Delphi  study  is  not  necessarily  a  good 
measure  of  the  technique’s  success.  Furthermore,  the  reaching  of  a  consensus  or  central 
theme  is  also  not  a  measure  of  Delphi  success.  Turoff  explains  some  approaches  that 
could  go  far  in  reducing  the  number  of  necessary  Delphi  rounds,  beginning  with  the 
researcher’s  dedication  of  considerable  time  and  effort  to  the  formulation  of  obvious 
issues  that  are  clearly  exhibited  [Turoff,  1975:  89].  A  second  approach  is  to  ask  the 
expert  participants  for  both  their  positions  on  an  item  and  their  underlying  assumptions — 
in  the  very  first  round  of  the  study  [Turoff,  1975:  89].  These  approaches  could  result  in  a 
2-3  round  Delphi  study  that  holds  extreme  value  to  the  researcher.  As  for  reaching  a 
consensus,  it  is  not  necessary  and  may  not  be  desirable  in  some  situations.  Delphi  studies 
can  result  in  divergence  of  ideas,  the  polarization  of  experts  into  two  camps.  This  in 
itself  is  valuable  insight  into  the  applicable  issues  that  could  make  a  given  Delphi  study 
quite  fruitful.  However,  in  the  case  of  this  research  aimed  at  creating  a  decision  support 
tool  that  is  universally  applicable  to  the  entire  USAF,  a  group  consensus  was  desired. 
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The  Delphi  method  is  a  well-tested  and  appropriate  approach  to  efforts  geared 
toward  generating  a  consensus  [Linstone  and  Turoff,  1975:  75].  In  the  case  of  this 
research,  the  extrinsic  decision  factors  affecting  force  mix  selection  were  already  gleaned 
through  CDM  interviewing  and  content  analysis.  It  was  the  determination  of  the  best 
relationships  of  these  factors  within  the  candidate  decision  support  tool  that  required  a 
method  of  access  to  individual  expertise  and  judgements  that  had  not  yet  been,  and  could 
not  be,  provided  for  by  basic  research  interviewing.  In  this  light,  the  Delphi  technique 
was  the  most  appropriate  method. 

Having  established  the  background  for  this  research  and  the  elicitation  methods 
used,  Chapter  III  discusses  the  Decision  Analysis  field  and  the  selection  of  the  most 
appropriate  Decision  Analysis  technique  for  this  decision  situation. 
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III.  Determining  the  Best  Decision  Analysis  Model 


Decision  Analysis 

This  chapter  discusses  the  progression  of  decision  analysis  applications  from  their 
basest  forms  to  more  complex  approaches  that  deal  with  uncertainty,  risk,  sequentiality, 
and  multiple  criteria  and  objectives. 

Structuring  a  Decision.  According  to  Derek  W.  Bunn,  decision  analysis  (DA)  is 
conventionally  understood  to  fall  within  the  larger  study  of  the  entire  management  field, 
with  a  primary  focus  on  choosing  a  sound  alternative  from  among  competing  potential 
alternatives  using  a  perspective  that  is  essentially  methodological  [Bunn,  1984:  1]. 

A  fundamental  philosophy  of  DA  is  the  breaking  down  of  a  decision  into  its 
component  parts.  An  otherwise  overwhelmingly  complex  problem  can  then  be  astutely 
analyzed  piece-by-piece,  with  the  subsequent  re-composition  of  the  problem  providing 
insights  and  recommendations  that  allow  the  decision  maker  to  attack  the  overall 
situation  with  more  confidence  and  competence  [Bunn,  1984:  4,  and  Clemen,  1990:  4-5]. 
The  pattern  of  relationships  developed  by  breaking  a  decision  problem  down  come  to 
constitute  a  particular  decision  model  [Bunn,  1984:  4], 

The  situational  features  that  can  make  a  decision  problem  sufficiently  complex  to 
warrant  DA  techniques  include  uncertainty,  multiple  objectives,  multiple  options,  and 
sequentiality  [Bunn,  1984:  4].  These  features  are  discussed  in  greater  detail  periodically 
throughout  the  chapter. 

In  first  approaching  a  decision  situation,  decision  makers  often  begin  with  an 
elementary  framework,  “taught  throughout  private  industry,  schools,  and  even  the  Girl 
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Scouts,”  that  includes  1)  defining  the  problem  2)  gathering  facts  and  making 
assumptions  3)  developing  possible  solutions  4)  analyzing  and  comparing  possible 
solutions  5)  selecting  the  best  solution  [Lemire,  1991:  5].  This  framework  constitutes  a 
decision  model  in  its  most  basic  form,  and  serves  as  the  sequence  flow  for  DA  processes. 

Robert  T.  Clemen  refines  and  expands  upon  this  basic  DA  sequence.  In  his  book, 
Making  Hard  Decisions,  Clemen  describes  a  step-by-step  process  that  also  begins  with 
identifying  the  problem  [Clemen,  1990:  5].  His  second  step  is  the  identification  of 
objectives  and  alternatives.  He  refers  to  the  third  and  fourth  steps,  decomposing  and 
modeling  the  problem  and  choosing  the  best  alternative,  respectively,  as  the  “modeling 
and  solution  [which]  form  the  heart  of  most  textbooks  on  decision  analysis”  [Clemen, 
1990:  6-7].  While  many  discussions  of  the  decision  making  process  stop  at  Clemen’s 
fourth  step,  Clemen  adds  more.  Sensitivity  analysis  should  be  performed  once  a  solution 
is  selected,  and  this  should  be  followed  by  a  determination  of  whether  further  analysis 
might  be  necessary  before  implementing  the  proposed  solution  [Clemen,  1990:  7]. 

Figure  2  depicts  a  flow  chart  representation  of  Clemen’s  decision  analysis 
process.  The  process  serves  as  a  sequential  guide  to  analyzing  decision  problems  in 
almost  any  managerial  arena.  However,  such  flow  chart  guides  are  woefully  inadequate 
in  that  they  leave  the  decision  maker  to  rely  on  his  or  her  own  capabilities,  experience, 
education,  and  resources  in  the  accomplishment  of  the  steps  of  the  process.  More 
advanced  DA  models  provide  many  of  those  tools  so  essential  to  decision  makers  in 
addressing  complex  decisions. 
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Figure  2:  Decision  analysis  process  flow  chart 
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The  framework  established  by  DA  flow  charts  like  that  shown  in  Figure  2  serve  as 
the  foundation  for  more  comprehensive  decision  support  tools  that  subsequently  break 
decision  situations  into  their  component  parts. 

Howard  and  Matheson,  in  their  book  The  Principles  and  Applications  of  Decision 
Analysis,  address  the  “heart  of. .  .decision  analysis”  (Clemen’s  decomposing  and 
modeling  the  problem  and  choosing  the  best  alternative)  with  an  analysis  flow  diagram 
that  forms  the  basis  for  creating  a  decision  support  tool  [Howard  and  Matheson,  1983: 

26].  Figure  3  presents  this  approach. 


Figure  3:  The  decision  analysis  cycle 


This  DA  cycle  provides  for  the  structuring  and  analysis  of  those  decision 
problems  that  are  sufficiently  complex  to  warrant  decomposition;  namely,  those  that 
include  probabilities,  uncertainty,  risk,  and  sequentiality — as  illustrated  by  the  phases 
shown  in  Figure  3. 
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Extending  Decision  Structures:  Probability ,  Uncertainty  and  Risk.  Decision  trees 
and  influence  diagrams  provide  decision  makers  with  useful  means  for  decomposing 
decision  problems  that  have  multiple  steps  (sequential)  and  that  involve  measures  of 
probability  and  uncertainty. 

Influence  diagrams  help  decision  makers  conceptualize  all  of  the  elements  of  a 
situation,  providing  simple  graphical  representations  of  which  decisions  are  to  be 
considered,  what  uncertain  events  will  influence  intermediate  and  final  outcomes,  and 
what  values  are  to  be  placed  on  potential  outcomes  [Clemen,  1990:  34]. 

Decision  trees  go  further  than  influence  diagrams,  representing  the  specific 
decision  points  in  a  sequential  process  while  providing  choice  branches  for  the  decision 
maker  to  consider  and  probability  branches  that  represent  chance  outcomes  (events)  that 
might  follow  these  choices  [Clemen,  1990:  49].  “A  decision  tree  represents  the  possible 
unfolding  of  events  in  temporal  order”  [Heckerman  et  al,  1995:  2],  Together,  influence 
diagrams  and  decision  trees  comprise  DA  models  that  serve  as  tools  for  decision  makers. 
However,  many  decisions  require  substantially  greater  decomposition  or  an  altogether 
different  approach.  Often,  these  models  are  used  only  for  setting  the  basic  structural 
foundation  for  addressing  uncertainty  and  risk.  One  group  of  authors  explains  the  place 
of  uncertainty  in  decision  making: 

. .  .ambiguity  or  probabilism  intervening  among  elements  of  the  task 
environment.  Such  uncertainty  is  regarded  as  intrinsic  within  the  judgement  or 
decision  problem;  it  cannot  (at  the  time  the  judgement  or  decision  is  made,  at 
least)  be  eliminated,  and  it  exists  independently  of  the  judge  or  decision  maker’s 
(or  the  analyst’s)  representation  of  the  problem.  [Hammond  et  al,  1980:  193] 
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Using  decision  trees  as  their  maps,  decision  makers  can  apply  expected  values  and 
expected  utilities  to  the  elements  of  their  respective  situations  in  order  to  effectively 
account  for  uncertainty  and  risk. 

Expected  value  refers  to  the  value  of  an  uncertain  outcome,  computed  as  the 
probability- weighted  average  of  all  possible  outcomes  [Clemen,  1990:  178].  For 
example,  if  only  two  outcomes  are  possible,  x  and  y,  and  each  is  weighted  at  a  probability 
of  .5,  then  the  x  value  is  multiplied  by  .5  and  the  y  value  is  multiplied  by  .5  and  the  total 
is  summed.  Therefore,  if  x  equals  100  units  and  y  equals  200  units,  then  the  expected 
value  of  the  uncertain  outcome  would  be  .5(100)  +  .5(200)  =  150  units. 

A  decision  maker’s  attitude  toward  risk  in  the  face  of  such  uncertainty  is 
necessarily  important  to  the  development  of  a  realistic  decision  support  tool,  and  is  most 
often  captured  with  an  expected  utility  function.  Utility  functions  are  computed  using  the 
decision  maker’s  points  of  indifference  concerning  potential  outcomes,  and  can  identify 
the  decision  maker  as  either  risk  averse  or  risk  seeking  [Clemen,  1990:  376-377]. 

Decision  makers  can  create  decision  trees  that  utilize  expected  values  and 
expected  utilities,  using  branches  of  decision  nodes  to  correspond  to  alternatives  and 
branches  of  chance  nodes  to  correspond  to  possible  states  of  individual  variables 
[Heckerman  et  al,  1995:  2].  This  alone  can  sufficiently  decompose  many  complex 
decision  situations,  involving  uncertainty,  risk,  and  sequentiality,  into  points  of  analysis 
that  can  generate  highly  logical  decisions.  Perhaps  Judith  Lemire  says  it  most  succinctly: 

One  approach  to  uncertainty  is  the  expected  value  theory.  In  this  case,  one 
can  calculate  the  expected  value  of  the  lottery  by  multiplying  each  outcome  by  the 
probability  of  it  occurring  and  then  summing  the  results.  Without  accounting  for 
risk,  an  individual  should  be  indifferent  to  the  lottery  and  the  certain  receipt  of  the 
expected  value  of  that  lottery.  Many  of  us,  however,  display  risk  aversion,  where 
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we  prefer  the  certainty.  Some  individuals  are  risk  prone,  where  the  game  itself 
has  value.  [Lemire,  1991:  26] 

Often,  the  construction  of  decision  trees  that  represent  uncertainty,  risk,  and  all 
the  choice  and  chance  points  in  a  sequential  decision  situation  can  become  quite  complex. 
However,  such  models  can  still  fall  short  of  addressing  all  of  the  potential  complexities  of 
many  decision  situations.  These  additional  complexities  take  the  form  of  multiple 
criteria  and  multiple  objectives. 

Multiple  Criteria/Obiective  Decision  Structures.  Decision  situations  approach 
their  most  extreme  complexities  when  they  combine  uncertainty  and  multiple  criteria 
[Lemire,  1991:  8-9].  Decisions  lacking  certain  information  are  difficult  because  of 
potentially  misapplied  probabilities  and  complications  in  pinpointing  risk  attitudes. 
Furthermore,  such  decisions  can  have  negative  results  despite  the  soundness  of  the 
process  [Lemire,  1991:  8],  Multiple  objective  decisions  can  add  significantly  to  this 
difficulty  in  that  they  must  often  compare  “apples  to  oranges”  in  considering  possibly 
conflicting  goals  [Lemire,  1991:  8]. 

Derek  W.  Bunn  quotes  Milan  Zeleny,  “It  has  become  more  and  more  difficult  to 
see  the  world  around  us  in  a  unidimensional  way  and  to  use  only  a  single  criterion  when 
judging  what  we  see”  [Bunn,  1984:  82],  Multiple  objectives  introduce  complexities  that 
can  be  quite  challenging,  such  as  in  a  situation  where  a  combination  of  factors  has  an 
overall  synergistic  value  that  is  attained  by  the  peaking  of  all  of  the  factors  (far 
outweighing  the  sum  of  the  individual  factors),  or  in  a  situation  where  simultaneous 
success  for  all  of  the  objectives  is  not  necessary  in  order  to  optimize  overall  success 
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[Clemen,  1990:  474],  Multiple  objectives  are  fast  becoming  the  norm  in  DA.  Bunn 
points  out: 

. .  .many  decisions  in  the  public  sector  appear  to  involve  payoff 
assessments  in  more  than  just  monetary  values.  Siting  a  nuclear  power  plant,  for 
example,  involves  issues  of  safety,  health,  environment,  and  reliability  as  well  as 
cost.  We  must  therefore  look  at  decision  criteria  that  are  multi-dimensional,  that 
can  cope  with  the  many  conflicting  objectives  with  which  decision  makers  are 
sometimes  faced.  [Bunn,  1984:  82] 

There  exist  numerous  approaches  to  dealing  with  multiple  objectives  and  criteria, 
including  the  Kepner-Tregoe  method  and  Multi-Attribute  Utility  Theory  (MAUT),  both 
of  which  are  discussed  later  in  this  chapter. 

The  progression  of  decision  situations  and  DA  techniques  from  simple  to  complex 
is  presented  in  this  section  only  to  provide  the  setting  for  creating  a  decision  support  tool 
for  force  mix  selection  during  USAF  crisis  action  campaign  planning.  This  situation 
most  certainly  involves  the  evaluation  of  multiple  criteria.  It  is  important  to  note, 
however,  that  such  force  mix  selection  is  limited  to  a  single  decision  of  which  asset  set  to 
deploy  given  a  particular  state  of  nature  at  a  discrete  point  in  time.  Therefore,  this 
decision  support  model  need  not  allow  for  sequential  decision  making.  A  decision  tree 
format  is  neither  required  nor  appropriate. 

Classifying  the  DA  Approaches 

This  section  discusses  the  general  theories  of  decision  making  and  some  of  the 
general  classes  of  models  used  in  DA.  Although  this  literature  review  is  thorough,  it  is 
limited  and  is  by  no  means  exhaustive  of  the  subject.  DA  is  a  growing  field  of  study, 
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with  literally  thousands  of  works  on  the  subject,  of  which  the  review  of  all  is  beyond  the 
scope  of  this  research. 

General  Theories.  The  models,  tools,  and  even  the  basic  theories  surrounding  the 
science  of  decision  analysis  can  be  centered  either  prescriptively  or  descriptively — or  in 
combination  of  both.  Prescriptive  approaches  are  also  referred  to  as  rational  or 
normative,  telling  decision  makers  how  to  accomplish  their  task.  Descriptive  approaches, 
as  the  name  implies,  describe  the  processes  of  decision  making  [Oxenfeldt  et  al,  1978:  v- 
vi].  This  research  requires  the  development  of  a  prescriptive  decision  support  tool  that 
will  aid  campaign  planners  in  quickly  and  effectively  choosing  force  mixes  in  times  of 
crises.  Much  of  the  descriptive  requirements  of  such  decision  making  are  already 
provided  in  training  and  ample  USAF  and  DoD  instructions.  However,  the  descriptive 
approach  is  mentioned  here  because  descriptive  versus  prescriptive  is  the  initial  dividing 
point  in  establishing  the  classes  of  DA  methods. 

According  to  Oxenfeldt  et  al,  writings  on  decision  making  and  analysis  differ  by 
two  fundamental  orientations:  The  mathematical  and  statistical  orientation  that 
constitutes  statistical  decision  theory,  and  the  cognitive  decision  functioning  that 
constitutes  psychological  decision  theory  [Oxenfeldt  et  al,  1978:  vii]. 

Hammond,  McClelland,  and  Mumpower,  in  their  book  Human  Judgement  and 
Decision  Making,  provide  six  general  theoretical  approaches  to  the  study  of  decision 
making:  Decision  Theory,  Behavioral  Decision  Theory,  Psychological  Decision  Theory, 
Social  Judgement  Theory,  Information  Integration  Theory,  and  Attribution  Theory. 
According  to  the  authors,  the  first  three  approaches  are  seated  in  economics,  while  the 
last  three  are  rooted  in  psychology  [Hammond  et  al,  1980:  9]. 
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Decision  Theory  (as  specifically  described  and  limited  here  by  Hammond  et  al),  is 
that  work  most  closely  associated  with  Ralph  Keeney  and  Howard  Raiffa,  and  is 
concerned  with  choosing  among  alternatives  with  multiple  attributes  [Hammond  et  al, 
1980:  10].  The  parameters  of  choosing  one  alternative  over  another  are  the  probability  of 
the  occurrence  of  the  alternative  and  the  expected  utility  to  the  decision  maker 
[Hammond  et  al,  1980:  9],  placing  this  approach  under  the  statistical  decision  theory 
umbrella  of  Oxenfeldt  et  al. 

Behavioral  Decision  Theory  is  based  upon  the  central  idea  of  human  decision 
making  falling  short  of  true  rationality.  This  approach  describes  the  less  than  optimal 
behavior  of  the  decision  maker  [Hammond  et  al,  1980:  10].  It  is  a  descriptive  approach 
that  falls  within  the  economics  sphere  but  not  necessarily  within  statistical  decision 
theory. 

Psychological  Decision  Theory  moves  one  step  beyond  description  of  human 
decision  making  and  moves  toward  explanation  and  prediction  of  decision  behavior 
[Hammond  et  al,  1980:  10].  It  does  not,  however,  prescribe  action  or  behavior. 

Social  Judgement  Theory  emphasizes  the  interaction  between  environment  and 
cognitive  systems,  having  its  origins  in  the  field  of  perception  [Hammond  et  al,  1980:  10- 
11].  Once  again,  its  study  is  descriptive  in  nature  and  clearly  falls  under  the 
psychological  decision  theory  umbrella. 

The  second  purely  psychological  approach  (fifth  approach  overall)  is  Information 
Integration  Theory.  “IIT  emphasizes  and  provides  for  the  analysis  of  the  cognitive 
integration  of  multiple  pieces  of  information  that  are  measured  subjectively,  and  for 
which  subjective  importance  is  also  measured”  [Hammond  et  al,  1980:  12],  This  theory 
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deals  with  the  situational  “cognitive  algebra”  employed,  such  as  the  averaging  of  factor 
values  in  some  cases  and  the  multiplying  of  factor  values  in  other  cases  [Hammond  et  al, 

1 980:  12].  Although  this  approach  is  descriptive  in  nature,  it  clearly  has  the  potential  to 
prescribe.  However,  the  approach  falls  under  descriptive  psychological  decision  theory. 

Finally,  Attribution  Theory  studies  causal  attribution  under  the  “psychology  of 
common  sense,”  directly  addressing  the  tension  between  common  sense  and  refined 
knowledge  [Hammond  et  al,  1980:  12].  Simply,  this  approach  addresses  the  question  of 
“when  will  a  person  rather  than  circumstances  be  blamed”  [Hammomd  et  al,  1980:  12], 
Jon  Doyle,  of  the  Massachusetts  Institute  of  Technology,  provides  another 
grouping  of  decision  making  theories.  He  refers  to  rational  decision  making  as  “choosing 
among  alternatives  in  a  way  that  ‘properly’  accords  with  the  preferences  and  beliefs  of  an 
individual  decision  maker  or  those  of  a  group  making  a  joint  decision”  [Doyle,  1998:  1]. 
The  subject  is  developed  under  six  general  theories:  Decision  Theory,  Decision  Analysis, 
Game  Theory,  Political  Theory,  Psychology,  and  Economics  [Doyle,  1998:1]. 

Doyle  discusses  Decision  Theory  and  Decision  Analysis  interchangeably.  Game 
Theory  is  sequential  by  nature,  involving  premeditated,  strategic  actions  and  cognitively 
contrived  responses,  and  so  is  not  applicable  to  this  research.  Political  Theory, 
Psychology,  and  Economics  are  not  expounded  upon  by  Doyle  [Doyle,  1998]  and  are 
considered  here  as  descriptive  schools  concerning  decision  making;  they  necessarily 
cross  over  to  statistical  decision  theory  when  attempting  prescription. 

Illustration  of  the  basic  separation  of  classes  and  general  theories  of  decision 
making  is  important  to  this  research  in  developing  the  most  appropriately  structured  and 
oriented  decision  support  tool  in  aiding  campaign  planners. 
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Force  mix  selection  decisions  require  prescriptive  tools  that  fall  within  statistical 
decision  theory.  Theories  and  studies  concerned  with  describing  the  psychology  of 
decision  making  offer  little  value  within  the  scope  of  this  thesis.  The  remainder  of  this 
chapter  discusses  methods  and  models  that  fall  within  statistical  decision  theory,  the  most 
appropriate  candidates  for  this  research  problem. 

Classes  of  DA  Methods.  The  classes  of  DA  methods  are  not  always  exclusive. 
Review  of  the  previous  section  will  show  that  some  models  can  fall  under  several 
theoretical  umbrellas.  Operations  research  and  psychology  are  the  primary  fields  of 
study  concerned  with  DA,  and  DA  methods  overlap  between  the  two  fields.  Figure  4 
provides  some  examples  to  illustrate  this  relationship  [Klimack,  2000]. 


DECISION  MAKING 


Figure  4:  Sample  of  decision  making  approaches 
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Having  already  established  the  need  in  the  USAF  for  a  prescriptive  (rational), 
statistical  decision  theory- based  decision  support  tool  for  campaign  planners  in  selecting 
force  mixes,  the  reader  can  readily  grasp  the  sphere  of  influence  this  research  falls  under 
by  reviewing  Figure  4.  Some  examples  of  rational  and  descriptive  DA  methodologies  are 
shown  in  Figure  5. 


Methodologies 


Rational  (Prescriptive) 


Descriptive 


Multi-Attribute  Utility  Theory 
Analytical  Hierarchy  Process 
Multi-objective  Linear  Programming 


Psychological  Decision  Theory 
Social  Judgement  Theory 
Prospect  Theory 


Figure  5:  Rational  and  prescriptive  DA  methodologies 


Michael  R.  Klein  and  Leif  B.  Methlie,  in  Knowledge-Based  Decision  Support 
Systems,  offer  what  might  be  the  most  easily  discemable  break  down  of  DA  methods 
within  respective  fields  [Klein  and  Methlie,  1990].  Interestingly,  they  illustrate 
differentiated  groupings  of  techniques  within  individual  fields  of  study,  and  then  show  a 
natural  convergence  of  all  towards  a  common  master  product — a  complete  knowledge- 
based  decision  support  system  [Klein  and  Methlie,  1990:  6].  Figure  6,  borrowed  from  the 
Klein  and  Methlie  text,  illustrates  perfectly  this  grouping  of  methods  within  disciplines. 
Notice  that,  according  to  Klein  and  Methlie,  only  prescriptive  theories  lead  to  decision 
support  systems  (the  right  side  heading  at  the  top  of  the  figure),  and  a  decision  support 
system  is  what  this  research  strives  to  create. 
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BEHAVIORAL  SCIENCES/DESCRIPTIVE  THEORIES  PRESCRIPTIVE  THEORIES 


Figure  6:  The  scientific  background  of  DA  and  support  systems 
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In  Figure  6,  the  acronym  DSS  stands  for  “decision  support  system,”  and  KB-DSS 
stands  for  “knowledge-based  decision  support  system.”  This  research  strives  to  create  a 
decision  support  tool  for  campaign  planners  using  techniques  described  along  the  right 
side  of  Figure  6,  namely  utility  (or  value)  functions,  maximizing,  modeling,  and 
optimizing,  among  others. 

With  earlier  sections  having  provided  ample  discussion  of  simple-to-complex  DA 
situations  and  some  of  the  general  theories  in  the  field,  the  right  side  of  Figure  6  clearly 
narrows  the  direction  of  this  thesis  in  satisfying  research  questions  3  and  4.  This  is 
vitally  important  because  much  of  this  entire  thesis  effort  is  the  search  for  the  most 
appropriate  method  for  modeling  the  situation. 

Specific  Approaches.  There  are  many  candidate  DA  methods  for  modeling  this 
particular  decision  situation.  The  most  appropriate  method  must  be  more  than  simply  the 
most  convenient,  the  simplest,  or  the  easiest  to  use.  Conversely,  it  need  not  be  the  most 
complex  nor  the  most  novel.  It  must  simply  be  the  most  appropriate  for  the  problem  of 
force  mix  selection  during  crisis  action  campaign  planning  in  the  USAF. 

Initially,  two  economics-rooted  decision  making  methods,  the  zero-sum  technique 
and  the  mixed  strategy  game  (and  game  theory  in  general),  are  discarded  as  candidates 
for  this  research.  These  techniques  use  the  concepts  of  changing  states  of  nature, 
sequentially-oriented  decision  structures,  and  competitor’s  actions  in  defining  logical 
“moves”  for  the  decision  maker  [Filippi  and  Nelson,  1984:  26].  Later  discussion 
addresses  the  characteristics  of  this  decision  situation  that  disqualify  these  concepts  as 
candidate  models. 
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Expected  value  and  expected  utility  (utility  theory)  were  touched  upon  earlier. 
Utility  theory  has  undeniable  applicability  to  this  decision  situation  in  that  potential 
decision  makers  in  this  case  possess  expertise  that  is  highly  subjective  in  nature  and  their 
decisions  will  necessarily  involve  subjective  factors.  Jon  Doyle  states,  “Though  the 
notions  of  preference  and  optimal  choice  have  qualitative  foundations,  most  practical 
treatments  of  decision  theory  represent  preference  orders  by  means  of  numerical  utility 
functions”  [Doyle,  1998:  3].  Utility  theory,  on  the  surface,  is  certainly  a  viable  candidate 
for  this  decision  situation,  given  this  researcher’s  goal  of  optimizing  force  mix  selection. 
Note  the  following  discourse: 

The  principle  of  maximizing  expected  utility  has  long  been  established  as 
the  guide  to  making  rational  decisions  (von  Neumann  and  Morgenstem  1947; 
Savage  1954;  Luce  and  Raiffa  1957).  It  rests  on  two  components:  probabilities 
for  representing  our  uncertainty  about  the  situation,  and  utilities  for  representing 
our  preferences.  [Chajewska  and  Roller,  1999:  1]. 

Despite  their  advantages  in  converting  human  judgement  and  preference  into 
quantitative  mathematical  measures,  utility  functions  can  be  problematic.  The  process  of 
acquiring  a  utility  function  from  an  individual  decision  maker  is  not  sufficiently 
understood  by  many  analysts,  and  “there  are  no  experts  to  ask  about  the  structure” 
because  every  person’s  utility  function  may  be  quite  different  [Chajewska  and  Roller, 
1999:  1].  Moreover,  the  elicitation  process  is  cognitively  difficult,  error  prone,  and  often 
time  consuming  [Chajewska  and  Roller,  1999:  1]. 

Another  drawback  to  basic  utility  functions  is  that  their  elicitation  takes  decision 
makers  away  from  reality  and  into  the  world  of  hypothetical  lotteries  [Goodwin  and 
Wright,  1991:  83].  A  decision  maker’s  attitude  towards  risk  may  be  quite  different,  given 
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a  real  situation,  than  that  presented  in  the  utility  function  that  had  been  elicited.  Even  if 
this  is  not  the  case,  utility  functions  can  result  in  contradictory  decisions  as  evidenced  by 
the  “Allais’s  paradox”  put  forward  in  1953,  in  which  the  existence  of  a  certain  outcome 
among  probable  outcomes  creates  decisions  that  are  inconsistent  with  the  established 
expected  utility  [Goodwin  and  Wright,  1991:  84-85]. 

Nonetheless,  utility  theory  is  especially  valuable  in  its  application  in  advanced 
forms  such  as  Multi-objective  Linear  Programming  (MOLP)  and  Multi-Attribute  Utility 
Theory  (MAUT).  MOLP  and  MAUT  serve  as  the  two  major  theoretical  approaches  to 
dealing  with  multiple  criteria  decision  situations  [Fuller  and  Carlsson,  1996:  140].  Basic, 
or  single  attribute,  utility  theory  is  not  applicable  to  this  research  because  this  decision 
situation  involves  a  number  of  potential  attributes  or  criteria  and  an  almost  infinite 
number  of  alternatives.  However,  both  MOLP  and  MAUT  surface  as  highly  viable 
candidate  models  in  this  case,  as  extensions  of  single  attribute  utility  theory  that 
encompass  multiple  attributes,  criteria,  and  objectives. 

This  chapter’s  discussion  of  decision  making  theory  and  different  DA  methods 
can  now  be  focused  clearly  on  multi-criteria  models  under  the  Operations  Research 
umbrella  of  approaches  to  DA.  In  addition  to  MOLP  and  MAUT,  there  are  many 
methods  for  addressing  multiple  criteria,  some  of  which  might  be  quite  appropriate  for 
this  decision  situation. 

Jack  Kloeber,  a  former  associate  professor  at  the  Air  Force  Institute  of 
Technology,  provides  a  break  down  of  some  common  multi-attribute  decision  analysis 
techniques,  presented  in  Table  1  [Kloeber,  1992:  16]. 
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Table  1 :  Breakdown  of  multiple  attribute  decision  making  tools 


CHARACTERISTICS 

SAWM 

MAUT 

MLR 

TOPSIS 

AHP 

Requires  linearity  of  response 

Yes 

No 

Yes 

No 

No 

Independence  of  alternatives 
required 

Yes 

No 

Yes 

Yes 

No 

Easy  to  do  sensitivity  analysis 

Yes 

No/Yes 

Yes 

No 

Yes 

Rank  reversal  possible 

No 

No 

No 

Yes 

Yes  | 

Criteria  importance 
determined  within  the  model 

No 

Yes  (VFT) 

Yes 

No 

Yes 

Scale  of  final  rankings  of 
alternatives 

Ordinal 

Unknown 

Interval 

Ordinal 

Unknown 

Ordinal 

Unknown 

Requirements  to  build  the 
model 

Criteria 
Weights 
Raw  values 

Entire  utility 
functions  for 
all 

Past  data 
and 

Ratings  of 
results 

Criteria 
weights 
and  ranges 

Pairwise 
comparison 
of  criteria 

Time  intensive? 

No 

Yes 

No 

No 

Sometimes 

Past  data  required? 

No 

No 

Yes 

No 

No  | 

Cognitively  plausible? 

w/right 

weights, 

simple 

problems 

Yes,  for 
simple 
problems 

No 

No 

No 

Importance  of  user  judgement 

Extremely 

important 

None 

None 

Scaling  problems 

Yes 

No 

Yes 

No 

No 

Consistently  and  easily 
combines  different  criteria 

No 

Yes 

No 

No 

Yes? 

Prescriptive  or  Descriptive? 

Prescript. 

Prescript. 

Descript. 

Prescript. 

Prescript. 

Table  1  includes  the  Simple  Additive  Weighting  Method  (SAWM),  MAUT  with 
VFT  (value  focused  thinking),  Multiple  Linear  Regression  (MLR),  Distance  from  Ideal 
(TOPSIS  for  example),  and  the  Analytic  Hierarchy  Process  (AHP).  This  listing  is 
certainly  not  exhaustive.  Other  multi-criteria  approaches  include  the  Kepner-Tregoe 
Method  (discussed  later),  the  Candidate/Critique  approach  (of  Linden  et  al,  also 
discussed  later),  and  Multi-objective  Linear  Programming  (MOLP). 

While  the  approaches  presented  by  Kloeber  in  Table  1  are  limited,  the  left  column 
of  his  table  presents  the  distinct  characteristics,  of  any  candidate  DA  approach,  that  are 
necessarily  addressed  in  determining  the  most  appropriate  decision  support  tool  for  this 


51 


decision  situation.  Comparing  the  characteristics  of  various  DA  methods  to  the 
requirements  of  this  decision  situation  generates  a  final  set  of  viable  candidate  DA 
models. 

Characteristics  of  This  Decision  Problem.  Campaign  planners  in  today’s  USAF 
require  a  decision  support  tool  that  captures  all  of  the  relevant  extrinsic  factors  and 
combines  them  into  a  quantitatively-oriented  aid  to  evaluating  candidate  force  mixes 
quickly  and  efficiently,  with  little  end-user  input  or  cognitive  deliberation.  Force  mix 
selection  during  crisis  action  planning  is  a  highly  critical  process  that  equates  to  a  non- 
repetitive,  executive-level  decision  of  possibly  overwhelming  magnitude. 

Force  mix  selection  is  a  non-sequential  decision  process  that  involves  multiple 
criteria  and  requires  a  prescriptive  (rational)  support  tool.  The  individual  values  or 
utilities  of  the  extrinsic  factors  that  affect  force  mix  selection  are  not  necessarily  linear 
[Buzo,  2000].  What’s  more,  user  judgement  is  extremely  important  in  this  case  because 
it  is  the  expertise  of  the  most  knowledgeable  campaign  planners  in  the  USAF  today  that 
should  be  incorporated  into  any  support  tool  that  hopes  to  produce  the  best  options  for 
campaign  planners  to  choose  from. 

Because  of  the  criticality  of  the  decisions  to  be  made,  this  decision  situation 
demands  that  the  importance  of  all  criteria  be  determined  within  the  support  tool,  and  that 
rank  reversals  among  different  criteria  are  not  possible  (discussed  in  greater  detail  later). 

To  recap,  the  requirements  of  this  decision  situation  are:  1)  the  model  must  be 
prescriptive,  2)  it  must  account  for  multiple  attributes,  3)  it  must  provide  for  non-linear 
criteria  values,  4)  user  judgement  is  extremely  important,  5)  criteria  importance  must  be 
determined  within  the  model,  6)  rank  reversals  cannot  be  allowed. 
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Inappropriate  Methods.  Before  addressing  those  multiple  criteria  DA  techniques 
that  are  not  appropriate  for  this  situation,  a  quick  review  of  other  disqualified  methods  is 
in  order.  Decision  trees  and  game  trees  are  not  appropriate  because  they  require 
sequentially-oriented  decision  structures  and/or  known  probabilities  for  chance 
occurrences,  notwithstanding  their  potentially  intractable  compositions  if  many  criteria 
are  evaluated.  Prakash  Shenoy  states: 

. .  .valuation  networks  are  more  appropriate  representations  than  decision 
trees  and  game  trees  for  symmetric  or  almost  symmetric  decision  problems 
involving  many  variables  (italics  added)  since  it  is  practically  intractable  to  either 
represent  or  solve  such  problems  using  either  decision  or  game  trees.  [Shenoy, 
1996:  2]. 

Also  excluded  as  candidates  are  the  zero-sum  game,  the  mixed  strategy  game, 
game  theory  in  general,  and  Bayesian  Decision  Criteria.  Bayesian  statistical  theory 
allows  for  decisions  based  upon  the  introduction  of  new  information  that  creates 
conditional  probabilities  [Bunn,  1984:  113-121],  More  specifically,  Bayes’  Theorem 
allows  for  revision  of  initial  probabilities  to  posterior  probabilities,  given  the  results  of 
either  a  previous  decision  or  an  experiment  [Hill  et  al,  1981:  143].  Even  allowing  for 
nonsequential  decision  making  based  upon  numerous  chance  occurrences  for  which 
there  exists  any  number  of  conditional  probabilities,  this  decision  situation  involves  no 
such  chance  occurrences  within  the  decision  process  itself. 

Obviously,  single  attribute  utility  theory  is  also  disqualified.  However,  utility 
theory  in  its  expanded  forms  (i.e.  MAUT)  remains  in  consideration. 

As  for  multiple  criteria  DA  methods,  the  single  additive  weighting  method 
(SAWM),  while  easily  applied  and  understood  by  decision  makers,  is  disqualified  as  a 
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candidate  model.  Its  most  important  disqualifying  characteristics  are  that  it  requires 
linear  value  measurements  for  criteria,  it  provides  no  method  for  evaluating  the  relative 
importance  of  criteria  within  the  model,  and  it  does  not  provide  an  avenue  for  converting 
expert  user  judgement  [Kloeber,  1992:  16]. 

Multiple  linear  regression  (MLR)  is  likewise  inappropriate.  MLR  also  requires 
linear  values  for  criteria.  More  importantly,  it  is  a  descriptive  model  that  requires  past 
data  to  generate  results  rather  than  the  expert  judgement  of  qualified  decision  makers, 
making  it  absolutely  inappropriate  for  a  decision  support  tool  that  attempts  to  use  expert 
judgement  in  its  formulation  [Kloeber,  1992:  16]. 

Distance  from  Ideal  (TOPSIS)  does  not  meet  the  requirements  of  this  decision 
situation  because  it  likewise  does  not  use  expert  judgement  in  its  computations. 
Additionally,  it  provides  no  measure  of  the  relative  importance  of  criteria  within  the 
model  while  allowing  for  rank  reversal  among  criteria  (the  importance  of  which  is 
discussed  later)  [Kloeber,  1992:  16]. 

Methods  Considered.  The  multi-attribute  DA  method  selected  to  create  the 
decision  support  tool  for  this  research  must  provide  an  avenue  for  quantifying  the  value 
that  campaign  planners  in  the  USAF  place  on  each  of  the  individual  extrinsic  factors 
affecting  force  mix  selection.  This  is  necessary  to  provide  for  computing  the  overall 
worth  of  any  candidate  force  mix — the  overriding  goal  of  the  MR-VAT  and  this  research. 
The  primary  challenge  of  creating  a  decision  support  tool  for  force  mix  selection  is 
quantifying  the  multiple  values  and  worth  assessments  of  the  decision  makers.  James  R. 
Miller  III  states: 
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The  concept  of  worth  is  a.  property  of  human  beings.  It  is  part  of  their 
conscious  perceptual  apparatus.  Human  beings  may  formulate  notions  of  worth 
by  observing  external  objects  and  activities  and  by  considering  situational 
circumstances;  and  they  may  impute  or  project  these  notions,  once  formulated, 
onto  the  external  objects  and  activities  being  assessed;  but  the  real  locus  of  worth 
remains  within  the  subjective  minds  of  human  decision-makers. 

[Miller,  1970:  14]. 


Miller  claims  that  “any  assessment  of  worth  is  by  definition  a  subjective  process” 
[Miller,  1970:  14].  This  researcher  agrees.  What  is  required  is  a  decision  support  tool 
capable  of  quantifying  this  subjectivity  and  doing  so  for  multiple  attributes. 

The  chosen  multi-attribute  method  must  also  allow  for  the  integration  of  all 
factors  into  an  overall  value  that  can  be  maximized  according  to  the  decision  maker’s 
preferences.  In  common  practice,  decision  makers  often  examine  only  parts  of  their 
overall  decision  situations.  They  compare  (A)  to  (B)  only  on  price,  for  example,  under 
the  assumption  of  ceteris  paribus  (other  things  being  equal),  often  sub-optimizing  the 
overall  solution  [Hill  et  al,  1981 :  108-109].  Ceteris  paribus  is  a  “very  handy 
methodological  simplification”  but  one  that  should  be  limited  to  piece-meal  construction 
of  an  overall,  integrative  matrix  or  model  [Hill  et  al,  1981 :  109-113]. 

Five  multi-attribute  DA  techniques  stood  out  as  candidates  for  this  decision 
situation.  Each  is  discussed  in  detail  in  upcoming  sections.  The  Kepner-Tregoe 
approach  is  the  first  method  considered  and  is  perhaps  the  simplest.  It  warrants 
consideration  because  it  specifically  addresses  multiple  criteria  of  highly  subjective  value 
in  the  making  of  decisions.  It  provides  for  the  rank  ordering  and  weighting  of  “wants” 
and  a  go/no  go  standard  for  “musts”  [Filippi  and  Nelson,  1984:  81]. 
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The  second  method  discussed  is  the  Candidate/Critique  approach  in  which  an 
automated  problem  solver  interactively  presents  candidate  solutions  to  an  end-user  who 
critiques  those  solutions  in  accordance  with  his  or  her  values  [Linden  et  al,  1997:  67], 

The  third  approach  discussed  is  decision  optimization  via  Multi-objective  Linear 
Programming  (MOLP)  methods. 

The  fourth  approach  is  the  Analytic  Hierarchy  Process  introduced  by  Thomas  L. 
Saaty  in  1971.  This  method  provides  a  means  for  decisions  to  be  made  on  complex 
issues  by  simplifying  “our  natural  decision-making  processes”  [Saaty,  1982:  5].  The 
method  serves  to  capture  multiple  attributes  and  to  assign  quantified  values  to  subjective 
judgements. 

Finally,  Multi-attribute  Utility  Theory  (MAUT)  utilizing  Value  Focused  Thinking 
(VFT)  is  examined.  With  VFT,  utility  functions  are  replaced  by  value  functions,  with  a 
decision  maker’s  values  formed  hierarchically  in  the  model.  Hereafter,  MAUT  with  VFT 
is  referred  to  as  simply  VFT. 

The  Kepner-Tregoe  Method 

The  Kepner-Tregoe  method  was  introduced  by  Charles  H.  Kepner  and  Benjamin 
B.  Tregoe  in  1965  as  part  of  their  firm’s  professional  problem  solving  and  decision 
making  curriculum  [Kepner  and  Tregoe,  1965:  v-vi].  While  the  method  is  somewhat 
dated  and  surprisingly  simple  by  modem  DA  standards,  it  nonetheless  possesses  intuitive 
merit  in  dealing  with  highly  subjective,  multi-attribute  decision  situations. 

In  a  1984  thesis,  A  Heuristic  Approach  to  Decision-Making  for  the  Purchase  of 
Acquisition  Data.  David  Filippi  and  Richard  Nelson  claimed  that  “the  only  suitable 
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technique  which  utilizes  both  subjective  criteria  and  the  subjective  ratings  of  those 
criteria  is  the  Kepner-Tregoe  approach”  [Filippi  and  Nelson,  1984:  28].  While  such  a 
statement  is  in  error  given  the  work  of  Keeney,  Raiffa,  Saaty  and  others,  it  does  serve  to 
illustrate  the  usefulness  of  such  an  approach  to  the  decision  situation  at  hand. 

The  Kepner-Tregoe  approach  is  based  upon  seven  concepts  of  decision  making 
[Kepner  and  Tregoe,  1965:46-48]: 

1 .  Establish  the  objectives  of  the  decision 

2.  Determine  the  relative  importance  of  the  objectives 

3.  Develop  alternative  actions 

4.  Evaluate  these  potential  alternatives  against  the  established  objectives 

5.  Select  a  tentative  decision  (choice  that  best  achieves  the  objectives) 

6.  Explore  this  tentative  decision  for  possible  adverse  consequences 

7.  If  this  decision  becomes  final,  control  the  effects  of  the  adverse 
consequences  with  preventative  planning 

The  procedure  for  accomplishing  the  Kepner-Tregoe  method  follows  these  seven 
concepts  almost  exactly.  The  decision  maker  first  establishes  his  or  her  objectives. 
Determining  the  relative  importance  of  these  objectives  begins  with  a  separation  of  them 
into  “MUST”  and  “WANT”  categories.  A  MUST  objective  is  one  that  must  be  met  and 
cannot  be  compromised.  A  WANT  objective  is  one  that  can  generate  the  best  possible 
performance  from  a  decision,  but  which  can  be  bargained  [Kepner  and  Tregoe,  1965:  48]. 

The  MUST  objectives  are  placed  at  the  top  as  go/no  go  standards.  The  WANT 
objectives  are  then  rank-ordered  and  given  relative  weights,  either  as  multiples  of  the 
lowest-ranked  objective  or  on  a  scale  of  1  to  10  [Kepner-Tregoe,  1965:  48]. 
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The  decision  maker  then  develops  alternative  actions  and  evaluates  them  against 
the  MUST  and  WANT  objectives.  Obviously,  an  alternative  that  does  not  meet  all  of  the 
MUST  objectives  is  discarded.  The  decision  maker  then  values  (scores)  the  surviving 
alternatives  subjectively  against  each  WANT  objective  using  a  scale  of  1  to  10  or  1  to 
100,  with  each  alternative’s  value  multiplied  by  the  weight  of  the  corresponding  WANT 
objective.  These  products  are  then  summed  to  produce  an  overall  score  for  each 
alternative  [Kepner-Tregoe,  1965:  46-50]. 

The  highest  scoring  alternative  becomes  the  tentative  solution.  The  decision 
maker  is  not  done  there.  He  or  she  must  then  brainstorm  potential  adverse  consequences 
for  implementing  the  tentative  solution.  These  consequences  are  given  “probabilities”  of 
occurring  from  1  to  10  (odd,  but  the  scoring  system  requires  that  the  probabilities  be 
scored  at  greater  than  1)  and  the  decision  maker  again  subjectively  evaluates  the  value  of 
each  (negative  impact)  on  a  scale  of  1  to  10  [Kepner-Tregoe,  1965:  198-199].  These 
values  are  summed  and  the  total  is  subtracted  from  the  original  overall  score  for  the 
alternative.  Of  course,  this  may  reduce  the  tentative  solution’s  value  and  result  in  a 
different  alternative  becoming  the  tentative  solution,  and  the  process  is  repeated. 

The  final  step  is  to  plan  for  the  adverse  consequences  and  implement  preventative 
measures  that  may  mitigate  their  effects.  Figure  7  provides  an  example  of  the  Kepner- 
Tregoe  scoring  method.  In  the  figure,  notice  that  the  campaign  planner  is  evaluating  two 
candidate  force  mixes,  Mix  A  and  Mix  B,  each  with  a  distinct  set  of  fighter  and  bomber 
aircraft.  The  MUST  and  WANT  objectives  are  comprised  of  extrinsic  force  mix 
selection  factors. 
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Mix  A  Mix  B 

MUST  Objectives  Weight  Rating  Score  Rating  Score 

Host  Nation  Allow  Assets  In? 

N/A 

N/A 

Go 

N/A 

Go 

Runway  Length  and  Width  Sufficient  at  Loc.? 

N/A 

N/A 

Go 

N/A 

Go 

Mix  A  Mix  B 

WANT  Objectives  Weight  Rating  Score  Rating  Score 

Enemy  ISR  Capability 

9.6 

10 

96 

7 

70 

POL  Storage  Capacity 

8.7 

7 

60.9 

9 

78.3 

Distance  from  Mission  Targets 

7.8 

6 

46.8 

8 

62.4 

Ability  to  Re-Supply 

6.9 

7 

48.3 

9 

62.1 

Tota 

Score:  252  272.8 

Figure  7:  The  Kepner-Tregoe  scoring  method 


Independent  of  potential  adverse  circumstances  related  to  the  decision  that  must 
be  quantified  as  mentioned  earlier,  the  campaign  planner  would  select  Mix  B  using  the 
example  shown  in  Figure  7.  Notice  that  the  Kepner-Tregoe  method  provides  for  decision 
makers  to  establish  weights  for  the  WANT  objectives  prior  to  assessing  the  value  that 
each  alternative  has  against  each  objective.  The  method  allows  for  an  infinite  number  of 
alternatives  to  then  be  valued  against  an  existing  model  (i.e.  mixes  C,  D,  and  F  in  the 
example). 

This  method  is  well  suited  to  the  decision  situation  at  hand.  It  provides  for 
quantifying  otherwise  subjective  decision  factors  and  for  doing  so  for  multiple  criteria. 
Furthermore,  the  development  of  the  rank-ordering  and  weighting  of  the  MUST  and 
WANT  objectives  can  be  accomplished  using  the  expert  judgement  of  campaign  planners 
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across  the  entire  USAF.  However,  the  ratings  provided  by  the  decision  maker  in  valuing 
each  alternative  against  each  objective  must  be  done  independent  of  this  USAF- wide 
expertise,  during  crisis  action  planning  at  the  very  time  the  decision  must  be  made.  A 
better  decision  support  tool  would  allow  the  pool  of  representative  USAF  experts  to 
develop  the  value  of  the  alternatives  (against  the  objectives)  beforehand,  along  with  the 
rankings  and  weights  of  the  objectives.  That  would  lend  more  consistency  to  results. 

The  primary  goal  of  this  thesis  is  to  develop  a  decision  support  tool  that  will 
enable  selection  of  best  possible  force  mixes.  With  the  Kepner-Tregoe  method,  such  a 
support  tool  could  approach  this  goal,  but  would  not  reasonably  assure  its  attainment. 
Why?  Because  the  end-using  decision  maker  inputs  every  value  of  every  alternative  in 
complete  subjectivity.  With  Kepner-Tregoe,  consistency  could  be  almost  nonexistent. 
The  method  could  present  significant  sub-optimizations  if  a  given  campaign  planner 
lacked  the  knowledge  and  experience  to  judge  alternatives  against  objectives  at  the 
highest  level  possible.  This  researcher  posits  that  the  highest  level  possible  is  that 
expertise  provided  by  a  pool  of  the  most  expert  campaign  planners.  Additionally,  the 
end-using  decision  maker — even  if  the  most  qualified  in  his  or  her  profession — may 
require  considerable  time  and  cognitive  exertion  in  valuing  the  alternatives  at  the  very 
moment  when  time  is  critical.  Such  a  situation  runs  contrary  to  the  goals  of  this  decision 
support  tool. 

The  Kepner-Tregoe  method  is  a  marvelous  approach  when  the  end-using  decision 
maker  understands  the  alternatives  and  objectives  better  than  anyone  else,  or  when  that 
decision  maker  is  solely  responsible  for  every  aspect  of  the  decision  situation.  For  USAF 
campaign  planning,  we  need  something  more. 
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The  Candidate/Critique  Method 

The  key  concept  of  the  Candidate/Critique  model  of  problem  solving  is  that  it  is 
interactive.  Decision  makers  input  their  subjective  preferences  via  an  iterative  process 
with  an  automated  decision  support  tool.  This  method  is  one  of  the  latest  developments 
in  operations  research,  with  Thomas  and  Fischer  (1996),  McCalla  et  al  (1996),  and 
Mukhopadhyay  and  Mostafa  (1996),  all  paving  the  way  by  introducing  decision 
support/information  filtering  models  that  automatically  infer  user  preferences  via 
multiple  iterations  driven  internally  [Linden  et  al,  1997:  67].  Actual  Candidate/Critique 
methodology  goes  one  step  further. 

The  natural  extension  of  the  approach  is  that  of  Linden  et  al.  They  are  developing 
an  automated  decision  support  system  utilizing  a  manual  user  model  counterpart.  Their 
system  provides  for  progressive  dialogue  whereby  the  automated  problem  solver  provides 
available  options  to  the  user  and  the  user  provides  information  as  to  the  quality  of  those 
options  [Linden  et  al,  1997:  68].  A  key  is  that  the  system’s  user  model  improves  over 
time  and  ultimately  results  in  a  solution  that  is  acceptable  to  the  decision  maker  [Linden 
et  al,  1997:  68].  Figure  8  provides  an  example  of  a  basic  problem  solving  dialogue 
between  a  travel  agent  and  a  client. 

Figure  8  was  borrowed  from  the  Linden  et  al  paper,  Interactive  Assessment  of 
User  Preference  Models:  The  Automated  Travel  Assistant  [Linden  et  al,  1997:  68], 
Throughout  the  progressive  interaction,  the  agent  leams  more  and  more  about  the  client’s 
subjective  preferences  such  as  his  desire  to  fly  on  American  Airlines  and  his  moderate 
price  sensitivity.  An  important  consideration  is  that  the  client's  preferences  involve 
somewhat  complicated  tradeoffs  between  the  flight,  airline,  and  departure  times  that 
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could  become  intractable  if  modeled  ahead  of  time  within  the  automated  decision  tool 


[Linden  etal  1997:  68], 


Client:  “I  want  to  fly  from  Seattle  to  Newark  next  Tuesday  afternoon.” 

Agent:  “I’ve  got  a  United  flight  at  3:30pm  for  $500  and  an  American  flight  at  12:30pm 
for  $520.” 

Client:  “I  can’t  leave  before  3:00pm  but  I  do  prefer  American.” 

Agent:  “I  have  another  American  flight  through  Denver  at  4:00pm  for  $530.” 

Client:  “That’s  pretty  expensive.  I’d  be  willing  to  go  on  a  later  flight  or  another  airline 
if  it’d  be  much  cheaper.” 

Agent:  “The  cheapest  flight  is  USAir  at  8:00pm  for  $490.” 

Client:  “In  that  case,  the  American  flight  is  fine.” 


Figure  8:  Example  dialogue  between  a  travel  agent  and  client. 


The  Candidate/Critique  method  aims  to  replicate  this  iterative  process  between 
agent  and  client  as  closely  as  possible,  with  an  automated  problem  solving  tool  that  does 
not  calculate  subjective  values  ahead  of  time — it  calculates  them  from  the  user  model 
inputs  as  they  occur  [Linden  et  al,  1997:  69]. 

Although  we  do  not  attempt  to  implement  a  natural  language  interface,  we 
would  still  like  to  capture  the  essence  of  this  problem-solving  process.  In  these 
problems,  the  system  has  access  to  a  large  dataset  and  the  problem-solving 
methods  unavailable  to  the  user.  The  user  has  access  to  preference  information 
not  directly  available  to  the  system.  The  basic  mode  of  interaction  is  iterative  and 
cooperative,  where  the  system  and  the  user  both  attempt  to  convey  only  relevant 
knowledge.  The  problem  is  considered  solved  when  the  user  is  presented  with  a 
solution  he  considers  acceptable.  [Linden  et  al,  1997:  69] 


Two  significant  drawbacks  exist  with  the  Candidate/Critique  method  in  its  current 
early  stage.  First,  the  argument  that  computing  subjective  values  ahead  of  time  can 
become  intractable  is  countered  by  the  argument  that  any  such  computations  that  are  in 
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fact  manageable  should  be  included  ahead  of  time.  This  reduces  or  eliminates  the 
possibility  of  involving  the  end-using  decision  maker  in  an  intractable  process. 

Especially  in  cases  such  as  force  mix  selection  during  crisis  action  planning,  whereby  the 
expert  opinions  of  seasoned  campaign  planners  across  the  entire  USAF  can  present  a 
level  of  expertise  not  attainable  by  any  one  expert.  Interestingly,  the  Candidate/Critique 
system  uses  MAUT  as  its  underlying  user  model,  differing  from  traditional  MAUT  only 
in  that  the  system  creates  value  functions  at  the  time  a  decision  is  being  made.  While  this 
gives  a  level  of  reflexive  support  to  the  subjective  values  of  end-using  decision  makers 
that  rivals,  or  exceeds,  that  of  the  Kepner-Tregoe  method,  it  leaves  out  the  valuable  group 
expertise,  could  lead  to  significant  inconsistencies,  and  could  prove  most  time-consuming 
just  when  time  is  of  the  essence. 

The  potential  time-consuming  property  of  the  Candidate/Critique  approach  is  its 
second  major  drawback.  Depending  upon  the  problem  domain,  preferences  could 
“amount  to  an  explicit  total  order  over  all  candidate  solutions”  and  the  subjective 
critiques  could  become  “pairwise  comparisons  between  two  candidates  where  nothing 
more  could  be  inferred  about  the  user’s  preference  ordering”  [Linden  et  al,  1997:  70]. 

The  authors  admit  that  the  model  could  be  “intractable  and  unrealistic  in  its  full 
generality”  [Linden  et  al,  1997:  70].  They  suggest  its  use  for  graphical  layout  problems, 
internet  searches,  merchandise  selection,  and  airline  reservation  selection  [Linden  et  al 
1997:  69].  Force  mix  selection,  on  the  other  hand,  involves  a  myriad  of  extrinsic  and 
intrinsic  factors  that  combine  with  an  almost  unlimited  number  of  potential  fighter  and 
bomber  force  mix  combinations,  making  Candidate/Critique  unsuitable  for  now. 
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The  Candidate/Critique  method  has  tremendous  potential  in  the  campaign 
planning  arena  but  not  with  respect  to  the  extrinsically-oriented  decision  support  tool 
researched  here.  Extrinsic  factors,  and  intrinsic  factors  via  future  research,  should  be 
identified,  valued,  and  structured  into  a  viable  model  in  support  of  M-R  VAT.  This 
would  be  necessary  whether  or  not  Candidate/Critique  were  used,  since  that  method 
requires  MAUT  development  anyway  (a  complete  DA  technique  in  its  own  right). 

However,  the  vision  for  M-R  VAT  is  that  it  become  a  decision  support  model  that 
integrates  intrinsic  and  extrinsic  decision  factors  along  with  corresponding  resource  lift 
requirements  (and  availability)  and  actual  combat  sortie  projections.  Given  a  situation  of 
evaluating  just  these  few  subjective  tradeoffs,  the  Candidate/Critique  approach  could 
prove  most  valuable  as  the  user  interface  for  M-R  VAT.  As  for  the  extrinsic  model 
proposed  by  this  thesis,  Candidate/Critique  is  an  inappropriate  method. 

Multi-Objective  Linear  Programming 

Multi-objective  linear  programming  (MOLP)  is  an  extension  of  basic  linear 
programming  that  addresses  the  problem  of  competing  objectives  in  a  decision  situation. 
Linear  programming  is  essentially  a  solution  optimizing  technique  developed  in  the 
1940’s  by  George  B.  Dantzig  and  originally  known  as  the  “programming  of 
interdependent  activities  in  a  linear  structure”  [Lee,  1972:  15].  Prior  to  addressing 
MOLP,  it  is  important  to  discuss  the  structure  of  basic  linear  programs. 

Put  simply,  linear  programming  addresses  situations  in  which  a  decision  must  be 
made,  restrictions  or  constraints  are  placed  upon  the  available  alternatives,  and  a  primary 
goal  or  objective  is  present  [Ragsdale,  1998:  17-18].  With  linear  programming,  it  must 
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be  possible  to  express  all  of  the  functions  as  a  linear  combination  or  weighted  sum  of  the 
variables,  producing  the  general  form  shown  in  Figure  9  [Ragsdale,  1998:  23]. 


MAX  (or  MIN): 

+  C2X  2  +  ...  +  CfJXn 

Subject  to: 

a\\X\  +  a\2-^2  +  +  a\nX„  —  b\ 

aklX]  +  ak2X2  + ...  +  aknXn  >  bk 
cim,Xj  +  am2X2  + ...  +  a„,„Xn  =bn 

Xj  >  0  for  j  =  1,2  ...  n 

Where: 

c(.  =  coefficients  of  the  objective  function  (such  as  dollars) 

X j  =  decision  variables  (alternatives  to  be  changed) 

ay  =  numeric  coefficient  in  the  tth  constraint  for  variable  Xy 

b-t  =  constraints 

Figure  9:  Basic  linear  program  formula 


For  single  overriding  objectives,  computer  programs  such  as  Excel  Solver  can 
utilize  formulas  like  that  shown  above  to  generate  optimal  solutions  to  the  decision 
problem.  Since  linear  optimization  models  assume  that  the  “values  of  all  the  relevant 
parameters  are  known  with  certainty,”  the  DA  problem  solving  efforts  concerning  this 
technique  are  prominently  focused  on  finding  the  most  efficient  search  technique  [Joshi, 
1980:  311]. 

MOLP  eliminates  the  limitation  that  a  decision  situation  have  just  one  overriding 
objective;  it  provides  a  realistic  and  robust  approach  to  dealing  with  multiple  competing 
objectives  so  long  as  linearity  of  the  attributes  of  the  situation  is  maintained  [Anderson  et 
al,  1974;  Joshi,  1980;  Lee,  1972;  Ragsdale,  1998], 
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The  difference  with  MOLP  is  that  multiple  objective  functions  (such  as  the  MAX 
or  MIN  function  in  Figure  9)  can  be  used  to  generate  a  new  overall  objective  function. 
For  example,  if  a  decision  maker  desired  three  competing  successful  outcomes,  each  of 
these  outcomes  (objectives)  would  be  solved  independently  of  the  other  two  but  within 
the  same  model,  with  all  constraints  enforced  for  each  iteration,  providing  three  separate 
optimal  solutions.  These  would  serve  as  the  best  known  possible  outcomes  for  each 
objective,  giving  the  decision  maker  valuable  reference  points  for  his  or  her  judgement  of 
the  final  solution.  More  importantly,  these  three  optimal  solutions  would  then  become 
target  values  in  the  formulation  of  the  new  overall  objective  function. 

With  target  values,  the  decision  maker  need  only  minimize  the  maximum  percent 
deviation  (“MINIMAX”)  of  each  objective  from  its  target  value  [Ragsdale,  1998:  276- 
286].  This  method  also  allows  for  weighting  the  percent  deviations  to  reflect  the  decision 
maker’s  preferences  with  regards  to  each  objective’s  importance.  The  formula  to 
generate  target  values  would  look  much  like  that  in  Figure  9  but  with  three  objective 
functions  rather  than  one.  The  final  formula’s  objective  function,  using  target  values, 
would  look  similar  to  Figure  10. 

In  Figure  10,  the  w>.  ’s  refer  to  the  individual  weights  assigned  as  the  measure  of 
importance  the  decision  maker  gives  each  objective,  and  the  Tt ’s  refer  to  the  target  values 

obtained  by  optimizing  each  objective  separately  as  mentioned  in  the  previous  paragraph. 
By  implementing  a  computer-generated  search  technique  (such  as  branch  and  bound)  to 
generate  the  lowest  possible  maximum  weighted  percent  deviation  among  the  three 
objectives,  MOLP  presents  a  remarkable  approach  to  solving  multi-criteria  problems  that 
far  exceeds  the  cognitively-generated  solutions  of  human  decision  makers. 
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Figure  10:  MOLP  objective  functions 

In  using  MOLP,  a  decision  maker  can  be  assured  that  no  better  feasible  solution 
exists,  given  that  the  constraints  are  sound  and  accurate,  the  weights  for  the  objectives  are 
accurate  representations  of  the  decision  maker’s  values,  and  all  of  the  attributes  are  linear 
[Joshi,  1980;  Keeney  and  Raiffa,  1976]. 

However,  the  development  of  weights  for  each  objective  could  become  an 
exhaustively  iterative  process  for  the  decision  maker  as  he  or  she  adjusts  the  weights  to 
explore  solutions  that  do  not  necessarily  occur  at  the  comer  points  of  the  feasible  region 
[Ragsdale,  1998:  287], 

Another  approach  to  MOLP  is  offered  by  Madhukar  Joshi,  and  provides  for  the 
computer’s  search  technique  to  generate  the  weights  [Joshi,  1980,  341].  In  his  approach, 
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each  of  K  objective  functions  is  given  a  weighting  coefficient  without  an  expressed 
aspiration  level,  such  that: 

Z*=ZC»Afork=  1.2, ...,K 

j= 1 

With  nonnegative  weights,  W],W2,...Wk,  such  that: 

Wl+W2+...  +  Wk=  1 

Then  the  MOLP  is  formulated  as: 

MAX:  Z  =  fw„Zt  =  flWk(flCvXJ) 

k= 1  >t=l  7=1 

Subject  to:  <  (=  or  >)£>,, i  =  1,2,. ..,m 

>0,  j  =  1,2, ...,n 
^  >0,  k  =  1,2,...,K 

2X=i 

k= 1 


With  this  formulation,  the  user  or  decision  maker  does  not  specify  the  objective 
weights,  this  is  left  to  the  computer  [Joshi,  1980:  341],  Therefore,  decision  maker  input 
as  to  the  relative  importance  of  the  competing  objectives  is  nonexistent!  It  would  seem 
that  MOLP  gives  us  a  situation  whereby  the  decision  maker  either  manipulates  weights  in 
the  search  for  solutions  or  leaves  the  importance  of  each  objective  up  to  the  optimization 
of  the  computer.  However,  this  statement  is  only  partially  true. 

According  to  Robert  Fuller  and  Christer  Carlsson,  a  number  of  MOLP  models 
involve  utility  theory-based  tradeoffs  between  objectives  and  include  integer  and 
stochastic  variables,  as  well  as  methods  using  “ideal”  and  “reference”  points  [Fuller  and 
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Carlsson,  1996:  140].  These  would  seem  to  better  capture  human  subjectivity  without  the 
measure  of  interactive  play  required  of  traditional  MOLP.  This  research  posits  that  such 
models  blur  the  distinction  between  MOLP  and  more  developed  multi-criteria  decision 
making  (MCDM)  techniques  such  as  MAUT  and  VFT.  In  fact,  Fuller  and  Carlsson 
group  both  MAUT  and  MOLP  under  the  MCDM  umbrella  and  imply  that  MOLP 
becomes  a  true  MCDM  technique  when  it  is  interactive  and  involves  utility  functions 
[Fuller  and  Carlsson,  1996:  139-140].  This  crosses  the  threshold  of  linearity  of  attributes 
and  addresses  non-linear  criteria,  more  closely  approximating  MAUT.  MAUT  (using 
VFT)  is  discussed  later  as  the  fifth  candidate  approach  to  this  decision  situation. 

Horn  et  al  address  MOLP:  “Historically,  multiple  objectives  (i.e.,  attributes  or 
criteria)  have  been  combined  ad  hoc  to  form  a  scalar  objective  function,  usually  through  a 
linear  combination  (weighted  sum)  of  the  multiple  attributes,  or  by  turning  objectives  into 
constraints”  [Horn  et  al,  1993:  1].  Such  would  tend  the  model  toward  the  requirement  for 
linearity  of  its  attributes. 

Some  of  the  problems  associated  with  linearity  requirements  are  presented  by 
Madhukar  Joshi: 

. .  .let  us  pause  to  discuss  the  meaning  of  linearity.  An  LP  problem  has 
three  properties:  proportionality,  additivity,  and  divisibility.  Proportionality 
implies  that  the  amount  of  resources  used  up  in  performing  an  activity  is 
proportional  to  the  value  of  the  corresponding  action  variable.  It  also  implies  that 
the  payoff  associated  with  that  activity  is  proportional  to  the  level  of  that  activity. 

. .  .The  [linear]  model  does  not  allow  for  possible  fatigue  [for  example]  at  the  end 
of  the  day,  nor  does  it  consider  the  increase  in  workers’  speed  due  to  familiarity. 
[Joshi,  1980:  321-322] 

Additivity,  another  condition  of  linearity,  necessitates  that  the  objective  function’s 
value  is  obtained  by  adding  the  contributions  of  each  activity.  This  implies  that  sales  of 
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one  item,  for  example,  do  not  compete  with  the  sales  of  another.  If  this  is  not  true,  then 
the  additivity  and  thus  linearity  assumptions  are  violated  [Joshi,  1980:  322], 

Ralph  Keeney  and  Howard  Raiffa  add  “it  is  not  always  appropriate  to  condense 
an  r-tuple  of  benefits  (bv...,br )  by  means  of  a  simple  linear  weighting  rule”  such  as: 

wxbx  + ...  +  wfr  + ...  +  wrbr 

Nor  is  it  always  appropriate  to  use  suitably  chosen  nonlinear  transformations  such  as: 

wi£i(^)  +  -  +  w, £,(&,)  + ...  +  wrgr(br) 

[Keeney  and  Raiffa,  1976:  22] 

Although  MOLP  is  an  admittedly  sound  technique  in  many  respects,  it  does  not 
meet  the  requirements  of  this  decision  situation.  The  need  for  attribute  linearity  is 
unrealistic  in  the  arena  of  extrinsic  factors  affecting  force  mix  selection.  The  assemblage 
of  attributes  into  linear  objective  functions  and  constraints,  as  described  by  Horn  et  al, 
poses  a  threat  to  any  MOLP  model’s  accurate  representation  of  the  subjective  values 
offered  by  campaign  planning  experts.  Perhaps  Ralph  Keeney  says  it  best: 

[Because]  the  consequences  are  significant. .  .the  amount  of  time  devoted 
to  careful  study  of  the  appropriate  values  is  miniscule  relative  to  the  time  used  to 
address  other  aspects  of  the  decision  situation.  The  “objective  function”  may  be 
chosen  in  an  hour  with  very  little  thought,  while  several  person-years  of  effort  and 
millions  of  dollars  may  be  used  to  model  the  relationships  between  the 
alternatives  and  the  consequences  and  to  gather  information  about  those 
relationships.  Since  values  are  the  entire  reason  for  caring  about  the  problem,  it 
would  seem  reasonable  to  use  a  portion  of  those  resources  to  structure,  quantify, 
and  understand  the  relevant  values.  [Keeney,  1992:  130]. 


Surely,  any  quantification  of  extrinsic  decision  factors  into  a  decision  support 
tool,  if  optimization  is  a  goal,  should  allow  for  accurate  transformations  of  subjectivity  as 
well  as  for  an  objective  function.  As  mentioned  earlier,  MOLP  only  approaches  this  once 
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it  blurs  the  line  into  becoming  a  MAUT  technique.  Says  Keeney,  “the  objective  function 
should  be  a  measurable  value  function  constructed  so  that  the  differences  in  value  derived 
for  the  consequences  have  a  meaning  in  addition  to  the  fact  that  larger  numbers  indicate 
preferred  consequences”  [Keeney,  1992:  132], 

The  Analytic  Hierarchy  Process 

Thomas  L.  Saaty’s  Analytic  Hierarchy  Process  (AHP)  is  a  technique  that  enables 
managers  to  make  decisions  concerning  complex  issues  by  simplifying  and  expediting 
their  natural  cognitive  decision  making  processes  [Saaty,  1982:  5].  Like  many  DA 
methods,  AHP  breaks  down  a  complex  and  largely  unstructured  decision  situation  into  its 
component  parts.  AHP  then  differentiates  itself,  as  explained  by  Saaty: 

. .  .arranging  these  [component]  parts,  or  variables,  into  a  hierarchic  order; 
assigning  numerical  values  to  subjective  judgements  on  the  relative  importance  of 
each  variable;  and  synthesizing  the  judgements  to  determine  which  variables  have 
the  highest  priority  and  should  be  acted  upon  to  influence  the  outcome  of  the 
situation.  [Saaty,  1982:  5]. 

More  important  to  the  decision  situation  of  this  research,  AHP  presents  a  highly 
effective  structure  for  group  decision  making  by  imposing  discipline  upon  the  thought 
processes  of  the  group  [Saaty,  1982:  5]. 

AHP’s  two  foundational  constructs  are  pairwise  comparison  among  multiple 
attributes  and  a  hierarchic  representation  of  attributes.  It  is  based  on  three  fundamental 
theories:  1)  The  human  mind  is  quite  capable  of  making  accurate  comparisons  between 
two  attributes  2)  All  important  problems  are  essentially  multi-criteria  with  an  inherent 
hierarchical  structure  3)  The  pairwise  comparisons  can  be  consistently  combined  within 
the  hierarchical  structure  to  yield  the  best  alternative  [Kloeber,  1992:  3]. 
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The  AHP  process  is  carried  out  in  four  steps.  First,  the  problem  is  set-up  into  a 
hierarchy  with  the  decision  objective  at  the  top,  attributes  of  the  decision  that  affect  its 
quality  in  the  middle,  and  the  alternatives  at  the  bottom  [Darko,  1987:  8].  The  second 
step  is  the  pairwise  comparison  of  the  middle  decision  attributes.  This  is  a  highly 
subjective  process  that  involves  the  use  of  a  constructed  value  scale  that  will  be  discussed 
shortly  [Darko,  1987:  9;  Kloeber,  1992:  8].  AHP’s  third  step  is  the  development  of 
priorities,  or  weights,  for  the  individual  elements  of  the  hierarchy.  Again,  this  is  rather 
subjective — which  is  desirable  for  this  decision  situation  as  well  as  many  others.  Finally, 
the  fourth  step  aggregates  these  weights  into  a  “vector  of  composite  weights  which  serve 
as  ratings  of  [the]  decision  alternatives. .  .in  achieving  the  most  general  objective  of  the 
problem”  [Zahedi,  1986:  99].  Simply  put,  step  four  ranks  the  alternatives  from  best  to 
worst  [Darko,  1987:  9]. 

A  scaling  technique  devised  by  Saaty  generates  consistency  and  flexibility  in 
quantifying  the  qualitative  attributes  or  criteria.  Numerical  ratios  for  qualitative  data  are 
made  possible  by  Saaty’s  Pairwise  Comparison/Importance  Intensity  Scale,  reproduced  in 
abbreviated  form  in  Table  2.  In  AHP,  even  numbers  are  used  for  intermediate  values. 


Table  2:  Saaty’s  Pairwise  Comparison  Scale 


Definition 

1 

Equal  importance  of  both  objectives 

3 

Weak  importance  of  one  element  over  another 

5 

Essential  or  strong  importance  of  one  element  over  another 

7 

Demonstrated  importance  of  one  element  over  another 

9 

Absolute  importance  of  one  element  over  another 
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Saaty’s  scale  is  sufficiently  robust  from  a  social  research  standpoint  [Saaty  and 
Vargas,  1982:  24],  and  provides  a  method  for  combining  pairwise  comparisons  in  a 
hierarchical  manner  that  allows  for  different  scales  to  be  used  for  different  attributes — 
generating  sufficient  flexibility  for  both  quantitative  and  qualitative  comparisons 
[Kloeber,  1992:  8], 

Two  considerations  in  evaluating  AHP  for  this  decision  problem  are  consistency 
and  coherence.  Coherence  is  addressed  shortly.  Consistency  for  a  decision  model  means 
that  it  must  capture  and  quantify  highly  subjective  criteria,  and  is  a  significant 
consideration. 

The  concept  of  consistency  is  best  shown  by  equation. 

If  A  =  2B 
and  B  =  2C 
then  A  =  4C 

As  the  equation  shows,  if  A  is  valued  or  preferred  twice  as  much  as  B,  and  B  is 
preferred  twice  as  much  as  C,  then  A  should  be  four  times  as  valuable  as  C.  If  A  were 
not  preferred  four  times  as  much  as  C,  then  that  comparison  would  be  considered 
inconsistent  [Luethke,  1987:  16]. 

AHP  provides  for  consistency  within  the  model  through  the  use  of  reciprocal 
matrices  and  calculating  the  weights  with  the  proper  eigenvector  (where  X  is  the  largest) 
which  is  approximated  by  the  geometric  mean  of  the  ratios  in  the  row  [Kloeber,  1992:  8- 
9].  Saaty  uses  the  consistency  ratio: 
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The  consistency  ratio  developed  by  equation  (1)  can  be  compared  to  the  matrix 
value  of  randomly  generated  numerical  judgements  (along  the  scale  of  1-9),  and  used  as 
the  indicator  of  consistency  of  the  decision  maker  in  preparing  the  pairwise  matrix 
[Kloeber,  1992:  8-9].  Consistency,  then,  is  properly  addressed  by  AHP.  However,  if 
many  criteria  affect  the  attributes  in  a  decision  situation,  or  if  there  are  many  attributes 
themselves,  then  the  pairwise  comparisons  become  intractable  due  to  the  number  (and  the 
size  of  the  matrix)  and  not  due  to  the  complexity  of  the  decision  [Kloeber,  1992:  17]. 

Coherence  is  yet  another  matter.  Coherence  here  refers  to  the  rationality  of  the 
decision  support  tool,  best  expressed  by  the  popular  perpetual  money-making  machine 
example. 

In  this  example,  an  incoherent  person  claims  that  X  is  less  likely  than  Y  and  that 
Y  is  in  turn  less  likely  than  Z.  However,  he  then  claims  that  Z  is  less  likely  than  X. 
Considering  that  he  would  be  rewarded  for  the  occurrence  of  X,  he  would  prefer  the 
greater  chance  of  Y  and  so  pays  the  rational  person  to  substitute  Y  for  X.  The  argument 
is  of  course  repeated,  this  time  with  a  paid  substitution  of  Z  for  Y  because  the  incoherent 
person  thought  Z  more  likely.  Having  been  paid  twice,  the  rational  person  then  makes  a 
third  sum  of  money  by  substituting  X  for  Z  because  the  incoherent  person  thought  X 
more  likely.  This  leaves  the  incoherent  person  where  he  started,  and  the  rational  person 
stands  to  make  money  perpetually  unless  the  incoherent  person  changes  his  uncertainty 
evaluations  [Bunn,  1984:  11], 
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This  example  of  an  irrational  approach  is  important  because  rank  reversal  of 
alternatives,  when  other  alternatives  are  added  to  or  subtracted  from  the  AHP  model,  is  a 
common  complaint  [Kloeber,  1992:  2], 

Rank  reversal  here  refers  not  to  the  ranks  and  weights  of  the  criteria,  rather  it 
refers  to  the  reversal  of  the  rankings  of  the  alternatives.  In  some  situations  in  which  AHP 
is  accomplished — with  the  results  computed  and  a  final  ranking  of  alternatives  given — 
the  addition  or  removal  of  a  well-chosen  alternative  and  subsequent  re-implementation  of 
the  AHP  process  results  in  two  or  more  alternatives  changing  relative  ranking  [Kloeber, 
1992:  13].  Such  occurs  with  absolutely  no  changes  in  the  attributes  of  either  alternative 
(i.e.,  changing  of  weights  during  sensitivity  analysis)  [Kloeber,  1992:  13].  This  is 
circular,  incoherent  reasoning  in  which  the  addition  of  Z  changes  one’s  preference 
between  X  and  Y. 

How  does  this  happen?  The  geometric  means  are  dependent  upon  all  of  the 
pairwise  comparisons.  If  any  are  deleted  or  added,  it  is  very  possible  that  the  mean  will 
change  as  will  the  weight  assigned  to  that  attribute  [Kloeber,  1992:  13].  More 
importantly,  this  will  occur  in  every  criteria  matrix  of  that  model  [Kloeber,  1992:  13]. 

Rank  reversal  does  not  make  AHP  invalid.  The  method  could  work  in  many 
situations  and  probably  holds  as  a  “descriptive  model”  [Kloeber,  1992:  13].  However, 
rank  reversal  in  contingency  planning  could  be  detrimental.  Campaign  planners  need 
best  force  mixes,  and  rank  reversals  place  in  serious  doubt  the  ability  of  AHP  to  deliver. 

It  is  therefore  disqualified  as  a  candidate  approach. 


75 


Multi-Attribute  Utility  Theory  and  Value  Focused  Thinking 

MAUT  has  become  a  prominent  recent  contributor  to  decision  analysis,  yielding  a 
“rigorous  technique  for  combining  attributes  multiplicatively  (thereby  incorporating 
nonlinearity),  and  for  handling  uncertainty  in  the  attribute  values”  [Horn  et  al,  1994:  1]. 

Value  Focused  Thinking  (VFT)  is  a  modification  of  MAUT  [Klimack,  2000] 
developed  by  Ralph  Keeney  that  differs  in  two  primary  ways:  its  treatment  of 
nonlinearity  of  model  inputs  (discussed  later),  and  its  focus  on  a  problem’s  values 
(objectives)  rather  than  its  candidate  alternatives. 

Stanley  Stafira  claims  that  DA  has  but  two  main  modeling  techniques:  alternative- 
focused  thinking  and  value-focused  thinking  [Stafira,  1995:  2-3  to  2-4].  Perhaps  a  better 
way  to  state  this  is  that  the  chosen  methodologies  of  any  DA  technique  can  be  either 
alternatives  driven  or  values  (objectives)  driven  [Parnell  et  al,  1998:  1338].  The 
objectives  driven  approach  starts  with  fundamental  objectives  and  underlying  sub¬ 
objectives  and  develops  quantifiable  attributes  for  the  lowest  level  objectives  up  through 
the  model,  while  the  alternatives  driven  approach  finds  the  alternatives  first  and  seeks 
identifiable  attributes  to  then  differentiate  these  alternatives  [Parnell  et  al,  1998:  1336]. 

Of  the  two  approaches,  the  objectives  driven  approach  is  more  appropriate  for  strategic 
situations  [Parnell  et  al,  1998:  1336].  Table  3  illustrates  the  differences  in  the  problem 
solving  steps  between  the  two  approaches  [Parnell  et  al,  1997:  7]. 

A  constructed  value  hierarchy  is  the  cornerstone  of  the  VFT  technique  [Buzo, 
2000:  21],  Value  hierarchies  serve  as  graphical  representations  of  all  of  the  relevant 
issues  needed  in  order  to  determine  an  overall  relative  value  for  each  candidate 
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Table  3:  Alternatives-driven  vs.  values-driven  models 


Step 

Alternative-Focused  Thinking 

Value-Focused  Thinking 

1 

Recognize  decision  problem 

Recognize  decision  problem 

2 

Identify  alternatives 

Specify  values 

3 

Specify  values 

Create  alternatives 

4 

Evaluate  alternatives 

Evaluate  alternatives 

5 

Select  an  alternative 

Select  an  alternative 

alternative  [Buzo,  2000:  21].  A  hierarchy  provides  for  information  collection, 
identification  of  alternatives,  facilitation  of  communication,  and  the  evaluation  of 
alternatives  [Kirkwood,  1997:  19-23]. 

A  value  hierarchy  can  be  constructed  from  the  bottom  up  by  identifying 
individual  organizational  tasks  or  sub-objectives,  grouping  them  together  with  the  use  of 
affinity  diagrams,  and  then  structuring  the  hierarchy  into  appropriate  tiers  [Parnell  et  al, 
1998:  1340], 

Value  hierarchies  constructed  from  the  top  down  are  referred  to  as  the  “gold 
standard”  [Parnell  et  al,  1998:  1338].  Top  down  constructions  are  based  upon  an 
organization’s  mission,  vision,  or  strategy,  with  the  objectives  and  tasks  broken  into 
progressively  lower  tiers  until  the  situation  is  wholly  encompassed.  These  constructions 
necessarily  involve  the  expertise  of  an  organization’s  top  decision  makers  and/or  ample 
organizational  guidance  as  to  the  factors. 
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Whichever  method  is  used  to  build  the  hierarchy,  Kirkwood  emphasizes  the  “test 
of  importance”  for  any  factor  whose  value  consideration  is  to  be  included.  The  test  is 
that  a  factor  should  be  included  only  if  potential  variations  among  the  candidate 
alternatives — with  respect  to  the  factor  under  consideration — could  change  the  preferred 
alternative  [Keeney,  1997:  19]. 

Once  a  hierarchy  is  constructed  to  model  the  decision  situation,  an  evaluation 
measure  is  assigned  to  each  objective  to  measure  its  degree  of  attainment,  such  as  “ higher 
salary  (the  evaluation  measure)  for  a  job  seeker”  [Kirkwood,  1997:  12].  Evaluation 
measures  serve  to  quantify  the  decision  maker’s  subjective  values.  VFT  requires  the 
normalizing  of  respective  evaluation  measures  whereby  the  analyst  does  not  use  (as  the 
measure  of  performance)  the  actual  evaluation  measure  score,  but  rather  the  “ proportion 
of  the  way  along  the  allowed  range  of  that  evaluation  measure  scale  where  the  score  for 
the  alternative  lies”  [Kirkwood,  1997:  57].  The  evaluation  measure,  then,  converts  a 
factor’s  scored  quantity  from  its  respective  units  into  a  common  set  of  units,  allowing 
many  criteria  to  be  combined  into  a  single  measure. 

VFT  incorporates  nonlinearity  through  the  functions  of  its  evaluation  measures, 
rather  than  through  a  model-wide  multiplicative  process.  Each  evaluation  measure,  upon 
elicitation  of  values  from  the  expert  decision  maker,  becomes  a  single-dimension  value 
function.  These  single-dimension  values  are  typically  exponential,  piecewise  linear,  or 
discrete  functions  [Clemen,  1996:  80;  Kirkwood,  1997:  62-68]. 

VFT  single-dimension  value  functions  convert  the  raw  graphical  representations 
of  candidate  alternatives  to  values  via  inverse  transformation  upon  the  value  graphs  of  the 
objectives  (i.e.,  an  exponential  curve).  Whichever  scale  is  used  for  the  values  (0  to  1,  0  to 
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10,  or  0  to  100)  must  be  used  consistently  for  all  values  within  the  model  [Kirkwood, 
1997:  61]. 

VFT  models  incorporate  relative  weights  for  the  single  dimension  value 
functions.  The  weighting  method  is  arguably  more  precise  than  that  of  the  Kepner- 
Tregoe  method  or  AHP.  A  number  of  weighting  methods  can  be  applied  [Kirkwood, 
1997:  68-70;  Clemen,  1996:  546-552],  of  which  the  “swing  weight”  method  advocated  by 
both  Kirkwood  and  Clemen  is  discussed. 

The  swing  weight  method  first  has  the  decision  maker  rank-order  all  of  the 
measures  and  then  study  the  ranges  of  each  of  the  measures  separately.  After  identifying 
the  ranges  of  swings  for  the  measures,  the  decision  maker  then  begins  pairwise 
comparisons  between  measures.  This  involves  the  decision  maker  determining  how 
much  swing  measure  A  must  have  in  order  to  match  the  value  of  B  when  B  is  swung 
from  its  lowest  to  its  highest  value  [Kirkwood,  1997:  71-72;  Clemen,  1996:  547], 

Equation  (2)  shows  that  A  need  swing  only  75  percent  of  its  range  to  equal  the  swing  of 
B  through  its  full  range. 


0.75A  =  B  (2) 

The  process  of  pairwise  comparisons  continues  until  there  is  one  less  equation 
than  the  number  of  objectives  [Buzo,  2000:  26].  Once  all  of  the  comparisons  are 
complete,  the  weights  are  normalized  to  sum  to  1 .  The  analyst  now  has  a  nearly  complete 
VFT  model  lacking  just  one  component — the  multi -objective  value  function.  The  most 
common  multi-objective  value  function  used  with  VFT  is  the  additive  form: 
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v(*,)  =  Zw/*v'*to) 

1=1 

Where  x  is  the  overall  value  objective,  x;  is  the  raw  score  of  attribute  i ,  v;  is  the 
single-dimension  value  function,  wt  is  the  weight  of  importance  on  attribute  / ,  and  n  is 

the  total  number  of  evaluation  measures. 

The  appropriateness  of  additivity  in  a  VFT  model  is  thoroughly  addressed  by 
Clemen,  Keeney,  and  Kirkwood  [Clemen,  1996:  579-580;  Keeney,  1992:  132-138; 
Kirkwood,  1997:  238-239].  The  requirement  for  an  additive  value  function  is  for  mutual 
preferential  independence  among  attributes.  One  should  note  that  careful,  proper 
selection  and  arrangement  of  objectives  within  a  value  hierarchy  can  almost  assure 
mutual  preferential  independence.  However,  formal  testing  should  be  accomplished. 

The  reader  is  encouraged  to  reference  Kirkwood’s  Theorems  9.19  and  9.20  in  addition  to 
Theorem  3.7  of  Keeney  and  Raiffa  [Kirkwood,  1997:  238-239;  Keeney  and  Raiffa, 

1976].  Clemen  also  offers  formal  testing  procedures  [Clemen,  1996:  580-582], 

VFT  is  certainly  a  viable  candidate  DA  method  for  this  highly  subjective  decision 
situation.  It  has  the  potential  of  accurately  converting  multiple  subjective  judgements 
into  quantifiable  value  measures  and  combining  them  in  a  manner  that  accounts  for  the 
nonlinearity  of  responses.  More  importantly,  it  does  so  with  a  focus  on  the  values  or 
objectives  of  the  decision  maker  as  the  driving  force.  This  is  important  because  an 
altematives-focused  method  would  limit  potential  solutions  to  a  pre-determined  and 
possibly  finite  set  of  choices.  VFT  leaves  the  decision  maker  free  to  create  or  select  any 
alternative  that  meets  the  requirements  generated  by  his  or  her  value  model. 
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The  Preferred  Method  for  this  Decision  Problem 

Following  a  background  review  of  DA  theories  and  methodologies,  and  a 
comparison  of  five  qualifying  candidate  approaches,  VFT  was  selected  as  the  method  for 
modeling  this  decision  situation.  The  efforts  of  this  research  validate  Buzo’s  selection  of 
VFT  for  the  original  modeling  of  this  subject  [Buzo,  2000]. 

In  his  model,  Buzo  incorporated  go/no  go  constraints  for  a  number  of  the  extrinsic 
decision  factors,  such  as  “Host  Nation:  Allow  Assets  In?  ”  [Buzo,  2000:  25].  This  is 
common  for  many  VFT  models  [Keeney,  1992;  Kirkwood,  1997].  However,  for 
inclusion  of  such  constraints  into  an  additive  hierarchy,  some  modification  within  the 
model’s  functions  is  necessary,  most  commonly  the  IF  statements  provided  by  most 
spreadsheet  applications. 

This  research  considers  the  possibility  of  creating  a  filter  model  for  the  VFT 
approach  used;  the  discussion  of  this  application  is  reserved  for  Chapter  IV.  A  filter 
serves  the  Go/No  Go  and  Go/No  Go  with  Optimization  rules  identified  by  Allan  Easton 
[Easton,  1980: 183-189].  The  Go/No  Go  Rule  provides  for  a  pre-testing  of  alternatives 
that  can  only  result  in  one  of  the  following  conclusions:  1)  one  alternative  will  survive 
the  test,  2)  no  alternative  will  survive,  3)  two  or  more  may  survive  [Easton,  1980:  183], 
This  rule  provides  for  pre-selection  of  only  those  alternatives  that  need  to  be  subjected  to 
further  evaluation  by  other  means  [Easton,  1980:  185], 

Easton  also  provides  for  single  attribute  optimization  following  a  comprehensive 
go/no  go  test  that  incorporates  all  of  the  other  attributes — the  Go/No  Go  with 
Optimization  [Easton,  1980:  187].  This  researcher  posits  that  a  filter  using  this  rule  could 
still  be  appropriate  in  the  case  of  VFT  because  the  VFT  model’s  multi-objective  value 
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function  could  serve  as  the  single  attribute  following  the  go/no  go  phase  of  other 
attributes.  Regardless,  Kepner-Tregoe  and  other  DA  techniques  (including  heuristic, 
cognitive  approaches)  allow  and  encourage  the  use  of  go/no  go  filters.  The  methodology 
of  the  research,  discussed  at  length  in  Chapter  IV,  outlines  the  process  for  considering  use 
of  a  go/no  go  filter  with  VFT. 
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IV.  Methodology 


Overview  of  the  Research:  The  Questions  and  Phases 

This  research  was  comprised  of  an  extensive  literature  review  and  multiple 
interviews  (in  person  and  via  email)  with  subject  matter  experts  in  an  effort  to  elicit  and 
extract  information  to  create  a  decision  support  tool  for  force  mix  selection  during  crisis 
action  planning. 

The  research  was  designed  to  answer  the  six  primary  research  questions  listed  in 
Table  4,  where  they  are  linked  to  their  respective  research  phases.  This  research 
addressed  the  questions  out  of  turn  because  research  question  5  pertained  initially  to  the 
selection  of  the  interview  participants,  and  research  question  3  was  best  addressed — via 
the  modified  Delphi  study — only  after  the  selection  of  the  most  appropriate  DA  method. 

The  four  research  phases  were:  I)  CDM  interviews  and  content  analysis,  II) 
literature  review  to  determine  the  best  DA  technique,  III)  modified  Delphi  interviews, 
and  IV)  development  of  the  final  decision  support  tool.  Table  4  shows  the  phases  and 
their  corresponding  research  questions. 


Table  4:  The  four  phases  of  research 


PHASE  I 

CDM  /  CONTENT  ANALYSIS 

QUESTIONS  1,  2,  5 

PHASE  II 

LITERATURE  REVIEW 

QUESTION  4 

PHASE  III 

ITERATIVE  INTERVIEWS  (DELPHI) 

QUESTIONS  3, 4,  5,  6 

PHASE  IV 

DEVELOPMENT  OF  MODEL 

QUESTIONS  3, 4,  5,  6 
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Phase  I:  CDM Interviews  and  Content  Analysis 

The  first  phase  addressed  research  questions  1,  2,  and  5.  The  first  research 
question  was:  Can  the  factors  affecting  force  mix  selection  in  response  to  theater  crises 
be  clearly  categorized  as  extrinsic  and  intrinsic,  and  can  these  categories  be  clearly 
defined?  The  second  question  was:  What  extrinsic  factors  are  important  to  campaign 
planners  when  selecting  aircraft  force  mixes  in  response  to  theater  crises?  And  the  fifth 
research  question  was:  Can  the  research  results  be  applied  USAF-wide? 

In  addressing  the  research  questions,  four  objectives  were  established  for  Phase  I. 
The  first  objective  was  to  clearly  distinguish,  if  possible,  what  constitutes  extrinsic  and 
intrinsic  force  mix  selection  decision  factors  in  the  eyes  of  the  subject  matter  experts — 
which  would  lead  to  definitions  for  each  type.  The  second  objective  was  to  identify  all  of 
the  extrinsic  factors  so  that  they  could  be  evaluated  as  contributors  to  the  decision  support 
tool.  The  factors,  to  lend  value  to  this  research,  would  necessarily  have  to  be  mutually 
exclusive  and  collectively  exhaustive  (MECE).  Establishing  factors  to  meet  these  criteria 
was  the  third  objective.  The  fourth  and  final  objective  of  this  phase  was  to  develop  and 
refine  an  accurate  definition  for  every  factor  identified.  Achievement  of  these  four 
objectives  assured  the  generation  of  sound  answers  to  questions  1  and  2. 

Content  analysis  of  the  appropriate  literature  was  the  first  step.  The  intent  during 
this  early  literature  review  was  to  include  every  factor  identified  and  then  to  add  any 
other  factors  identified  later  via  interviews.  Thus,  a  factor  would  be  included  if  either 
extracted  from  content  analysis  or  elicited  via  interview,  or  both — with  the  exception  of 
those  factors  identified  in  Buzo’s  thesis,  which  were  to  be  validated  [Buzo,  2000].  This 
provided  the  best  avenue  to  meeting  the  objective  that  all  extrinsic  factors  be  identified. 
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The  appropriate  literature  from  which  to  glean  decision  factors  on  this  subject  was 
the  official  guidance  that  is  provided  to  campaign  planners  throughout  the  USAF — the 
DoD  and  USAF  instruction  manuals  on  joint  planning  previously  discussed.  Buzo’s 
thesis,  being  a  compilation  of  extrinsic  decision  factors  into  a  single  decision  support 
tool,  contained  25  such  factors.  However,  because  this  research  sought  to  independently 
investigate  the  validity  of  those  factors,  they  were  not  included  in  the  results  of  the 
content  analysis. 

In  order  to  distinguish  between  extrinsic  and  intrinsic  factors,  examples  of  both 
were  extracted.  This  was  done  to  explain  the  differences  between  the  two  types  in  the 
minds  of  the  subject  matter  experts  who  were  to  be  interviewed  later.  This  in  turn  helped 
to  define  the  two  types  of  factors.  The  factors  extracted  via  content  analysis  are  shown  in 
Figure  1 . 

Once  the  collection  of  extrinsic  and  intrinsic  decision  factors  were  in  hand, 
interviews  with  the  experts,  using  the  Critical  Decision  Method  (CDM),  could  begin. 

CDM  is  an  interviewing  approach,  discussed  at  length  in  Chapter  II,  that  puts  the  focus 
on  those  most  critical  cognitive  points  or  tasks  of  a  job.  With  CDM,  “issues  can  be 
described  as  points  in  the  campaign  planning  process  where  critical  decisions,  decisions 
that  a  novice  would  have  sufficient  difficulty  overcoming,  may  be  required”  [Buzo,  2000: 
45].  It  is  these  critical  issues  that  could  hinder  timely  selection  of  the  best  force  mix. 

Selection  of  the  participating  experts  was  crucial  to  properly  addressing  research 
question  5,  Can  the  research  results  be  applied  USAF-wide?  Later  efforts  would  answer 
this  question  fully  but  this  was  the  necessary  first  step. 


85 


Included  as  interview  subjects  were  34  campaign  planning  experts  from  the  Air 
Staff  (HQ  USAF),  Air  Combat  Command  (ACC),  Air  Force  Southern  Command 
(SouthAF),  Air  Force  Special  Operations  Command  (AFSOC),  Air  Force  Central 
Command  (CentAF),  Joint  Forces  Command  (JFCom),  USAF  Fighter  Weapons  School, 
RED  FLAG,  USAF  Air  Expeditionary  Forces  Center  (AEFC),  USAF  College  of 
Aerospace  Doctrine  (CADRE),  and  the  AFSOC  Command  and  Control  schoolhouse 
(C2TIG).  Even  the  USAF  Reserves  were  included,  as  representatives  within  the  AEFC. 
This  highly  diversified  subject  pool  was  required  to  provide  a  broad  range  of  perspectives 
necessary  for  validity.  Scarcity  of  resources  precluded  interviews  with  expert 
representatives  from  overseas  commands.  However,  such  interviews  were  unnecessary 
because  those  commands  predominantly  employ  forces  during  conflict  rather  than 
selecting  which  force  mixes  to  deploy. 

Aside  from  the  necessary  representation  of  all  appropriate  organizations,  this 
research  required  a  wide  diversity  of  expertise  relating  to  operations.  The  subject  pool 
had  26  members  with  in-the-cockpit  operational  experience  with  a  wide  array  of  military 
aircraft  including:  F-15,  F-16,  F-4,  F-lll,  A-10,  C-130,  KC-10,  KC-135,  and  B-52.  Of 
the  34,  eight  were  career  logisticians.  Finally,  the  subjects  had  to  pass  the  following  base 
requirements:  minimum  grade  of  0-4  (or  civilian  equivalent),  minimum  of  12  years 
USAF  operational  experience,  minimum  of  two  years  of  campaign  planning  experience. 
All  subject  matter  experts  met  these  requirements. 

The  materials  used  for  the  CDM  interviews  were  borrowed  in  large  part  from 
Buzo’s  appendices  [Buzo,  2000:  100-1 12].  His  “bullet  background  paper”  was  modified 
to  meet  the  objectives  of  this  research  and  sent  via  email  to  all  participants  prior  to  the 
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interviews.  The  paper,  presented  in  Appendix  A,  provided  sufficient  background  to 
prepare  each  expert  for  the  task  at  hand. 

The  CDM  process  is  largely  unstructured  to  allow  experts  to  relay  completely 
their  own  personal  experiences.  However,  the  four  objectives  identified  earlier 
necessitated  a  semi-structured  approach  to  ensure  elicitation  of  only  appropriate  material. 
Buzo’s  original  interview  script  [Buzo,  2000:  102-104]  provided  the  necessary  structure. 
Again,  this  was  modified  (extensively)  to  meet  the  objectives  specific  to  this  research. 

The  modified  script,  seen  in  Appendix  B,  added  questions  pertaining  to  the  distinction 
between  extrinsic  and  intrinsic  factors,  the  assurance  of  MECE  among  factors,  and  the 
potential  definitions  for  extrinsic,  intrinsic,  and  each  of  the  individual  factors.  A  key 
difference  between  this  script  and  Buzo’s  is  that  this  script  aimed  to  elicit  all  of  the 
campaign  planning  factors  rather  than  just  the  extrinsic. 

The  materials  used  also  included  Buzo’s  list  of  extrinsic  factor  definitions  [Buzo, 
2000:  105-1 12]  and  a  listing  of  factors  extracted  via  content  analysis. 

The  procedure  for  conducting  a  CDM  interview  is  to  use  verbal  “probes” 
(questions)  to  elicit  critical  decision  points  from  the  subject  [Klein  et  al,  1997:  2; 

Militello,  1998:  1622].  The  probes  for  this  research  were  the  questions  (and  interviewer 
reminders)  printed  on  the  script. 

This  research  modified  the  CDM  procedure  used  by  Buzo  in  two  ways.  First, 
each  participant  was  asked  to  apply  any  scenario  with  which  he  or  she  felt  most  familiar.  • 
This  ensured  the  greatest  possible  diversity,  significantly  enhancing  the  applicability  of 
this  research  USAF-wide.  In  Buzo’s  research,  participants  were  limited  to  a  Southwest 
Asia  Theater  scenario  [Buzo,  2000:  51].  The  second  modification  was  to  introduce  each 
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participant  to  all  of  the  extrinsic  factors  that  had  been  extracted  via  content  analysis — but 
only  once  each  participant  had  exhausted  his  or  her  personal  list  of  factors. 

Definitions  were  elicited  for  each  factor  identified  by  each  participant.  Following 
this,  definitions  for  the  newly-introduced  factors  provided  by  the  researcher — and  agreed 
upon  by  the  participant — were  elicited.  Finally,  the  participants  compared  Buzo’s  factors 
and  definitions  to  those  that  had  been  elicited,  with  appropriate  adjustments  made. 

Phase  I  of  the  research  met  all  of  its  objectives,  properly  addressing  research 
questions  1  and  2,  and  providing  the  foundation  for  answering  research  question  5.  Pages 
15-19  of  Chapter  II  discuss  the  distinction  between  extrinsic  and  intrinsic  decision 
factors,  with  definitions  for  both  categories  presented  on  page  16.  The  final  set  of 
mutually  exclusive,  collectively  exhaustive  extrinsic  decision  factors  is  shown  in 
Appendix  C,  and  is  very  similar  to  Buzo’s  with  the  exception  of  the  factor  definitions. 

The  updated  definitions  are  presented  in  Appendix  D.  Further  discussion  of  the  results  of 
Phase  I  is  reserved  for  Chapter  V. 

Phase  II:  Literature  Review  to  Determine  the  Best  DA  Method 

The  objective  of  this  phase  was  to  determine  the  most  appropriate  DA  method  for 
modeling  the  decision  situation  and  creating  a  requisite  decision  support  tool. 
Accomplishment  of  this  single  objective  would  answer  research  question  4,  How  should 
these  factors  and  the  relationships  between  them  be  modeled? 

To  determine  the  most  appropriate  DA  technique,  it  was  necessary  to  first 
investigate  the  background  and  progression  of  DA  in  order  to  provide  for  the  selection  of 
a  method  based  on  a  foundation  of  knowledge  rather  than  just  simple  preference  or 
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convenience.  Once  the  comprehensive  background  was  established,  separation  and 
categorization  of  methods  into  families  of  techniques  and  schools  of  thought  could  be 
accomplished. 

After  categorizing  the  general  groups  of  DA  techniques,  an  analysis  of  the 
requirements  for  this  decision  situation  was  required  in  order  to  determine  which 
technique(s)  would  be  most  appropriate.  This  decision  situation  involves  multiple 
attributes,  severe  time  constraints,  critical  consequences,  non-sequentiality,  and  highly 
subjective  attribute  comparisons.  Discussion  of  this  was  presented  in  Chapter  III. 

The  characteristics  of  this  decision  situation  dictated  a  class  of  DA  techniques 
capable  of  measuring  multiple  attributes  and  quantifying  otherwise  subjective  criteria, 
among  other  considerations.  This  realization  narrowed  the  search  to  the  multi-criteria 
decision  making  (MCDM)  class  of  models  under  the  Operations  Research  umbrella. 

After  comparing  the  characteristics  of  this  smaller  set  of  candidate  techniques  to 
the  requirements  of  the  decision  situation,  Value  Focused  Thinking  (VFT)  was  selected 
as  the  most  appropriate  model.  This  in  effect  validated  the  work  of  Buzo  in  establishing 
his  decision  support  tool  [Buzo,  2000]. 

VFT  requires  the  assembling  of  criteria  into  a  value  hierarchy.  Therefore,  Buzo’s 
original  value  hierarchy  was  reproduced  exactly,  for  use  in  Phase  III  of  the  research 
[Buzo,  2000:  80].  In  addition,  another  candidate  hierarchy  was  constructed  and  used 
alongside  Buzo’s  in  the  next  phase.  This  hierarchy  resembled  Buzo’s,  but  with  a  go/no 
go  component  advocated  by  Easton,  Kepner,  and  Tregoe  [Easton,  1980;  Kepner  and 
Tregoe,  1965],  This  second  hierarchy  is  shown  in  Figure  11. 
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Figure  1 1 :  Candidate  VFT  hierarchy 
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Phase  III:  Iterative  Interviews  (Modified  Delphi)  to  Attain  Consensus 

Phase  III  was  structured  to  answer  research  question  3,  How  are  these  factors 
quantified  and  what  are  the  relationships  between  them?  In  addition,  it  laid  the 
groundwork  for  addressing  research  question  6,  How  can  the  now-quantified  extrinsic 
factors  be  effectively  combined  with  the  intrinsic  factors  to  develop  an  overall  asset  set 
utility  or  value? 

Four  objectives  were  established  for  this  phase.  The  primary  objective  was  to 
elicit  single-dimension  value  functions  for  each  of  the  extrinsic  factors,  in  accordance 
with  VFT  methodology.  The  second  objective  was  to  elicit  initial  relative  rankings  of 
importance  among  the  factors  in  order  to  seed  the  final  two  rounds  of  modified  Delphi, 
which  would  provide  the  weights  for  the  factors.  Evaluation  of  the  two  candidate  value 
hierarchies  was  the  third  objective.  And  the  fourth  objective,  perhaps  most  important  to 
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force  mix  selection,  was  to  begin  development  of  a  preliminary  theory  for  the  relationship 
between  extrinsic  value  and  intrinsic  value — in  the  eyes  of  the  experts — so  that  a 
mathematical  representation  of  such  could  be  developed. 

Phase  III  was  comprised  of  an  iterative  interviewing  and  email  questionnaire 
process  referred  to  here  as  a  modified  Delphi  study.  The  methodology  included  a 
comprehensive  first  round  of  face-to-face  interviews  that  focused  on  all  four  Phase  III 
objectives.  The  process  exhibited  Delphi  characteristics  in  that  all  participants  were 
guaranteed  anonymity,  the  process  involved  more  than  one  round,  and  the  results  were 
summarized  and  resubmitted  to  participants  between  rounds.  It  cannot  be  called  a  proper 
Delphi  because  one  could  argue  an  insufficient  number  of  rounds  and  because  the 
specific  summarization  techniques  advocated  by  Dalkey  and  others  were  not  incorporated 
[Dalkey,  1968].  The  second  and  third  rounds  of  the  modified  Delphi  were  in  the  form  of 
email  questionnaires  to  elicit  final  weights  for  the  factors. 

The  participant  selection  process  of  Phase  III  differed  from  that  of  Phase  I.  For 
the  iterative  interviews,  only  campaign  planning  experts  from  CADRE,  ACC,  AEFC,  and 
SouthAF  were  included.  ACC  and  AEFC  are  the  primary  brokers  of  asset  sets  in  support 
of  contingency  operations  from  the  continental  United  States  [Brown,  2000;  Broardt, 
2000],  and  CADRE  personnel  are  the  teachers  and  developers  of  basic  aerospace  doctrine 
for  the  21st  century.  Additionally,  because  Buzo’s  factors  and  model  had  thus  far  been 
validated,  his  CentAF-elicited  single-dimension  value  functions  and  factor  weights  were 
accepted  as  valid,  leaving  SouthAF  as  the  potentially  opposing  opinion  to  CentAF.  With 
proper  sensitivity  analysis  (discussed  later),  one  could  be  confident  that  value  functions 
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and  weights  elicited  from  these  four  agencies,  along  with  CentAF,  would  generate  a 
decision  support  tool  with  USAF-wide  applicability. 

The  inventory  of  materials  used  for  Phase  III  begins  with  the  list  of  extrinsic 
factors  constructed  during  Phase  I  along  with  Buzo’s  original  list  of  extrinsic  factors 
[Buzo,  2000],  Buzo’s  list  had  been  validated,  but  his  hierarchy  stood  unchallenged. 
Therefore,  the  shorter  list  compiled  during  Phase  I — which  had  some  of  the  factors 
combined  and  set  aside  as  a  go/no  go  filter  within  the  candidate  hierarchy — was  also 
evaluated  by  the  experts  during  the  iterative  interviews.  The  list  of  factor  definitions 
(Appendix  D)  was  an  integral  part  of  the  interview  materials.  Additionally,  both 
hierarchies  were  included  for  evaluation. 

The  materials  for  Phase  III  also  included  the  email  questionnaires  used  to 
determine  the  relative  weights,  discussed  later,  and  complete  sets  of  blank  single¬ 
dimension  value  function  elicitation  graphs,  discussed  here. 

Each  extrinsic  factor  has  its  own  single-dimension  value  function  within  a  VFT 
model,  therefore  each  extrinsic  factor  was  represented  by  a  blank  value  function 
elicitation  graph.  Figure  12  shows  such  an  example  graph  with  a  linear  value  function,  as 
might  be  elicited  from  a  decision  maker.  Notice  that  the  X-axis  is  reserved  for  the  range 
of  real  values  that  a  particular  factor  can  produce.  The  Y-axis  reflects  the  decision 
maker’s  transformed  value  given  that  a  factor’s  real  score  lies  at  some  point  along  the  X- 
axis.  This  Y-axis  value’s  range  is  from  0  to  1,  providing  for  the  normalization  of  all  such 
values  into  like  units  of  measurement  and  enhancing  the  objective  value  function’s 
overall  rating  of  a  candidate  alternative. 
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Figure  12:  Sample  single-dimension  value  function  graph 


The  procedure  was  identical  in  each  iterative  interview.  Each  interview  began 
with  participants  subjectively  evaluating  the  two  candidate  value  hierarchies.  These 
evaluations  were  unstructured  and  exploratory,  whereby  participants  were  free  to  sketch 
proposals  and  discuss  at  length  any  concerns  or  ideas.  Integral  to  the  development  of 
value  hierarchies  is  the  determination  of  parent  issues  and  sub-issues.  Note  that  parent 
issues  are  not  decision  factors,  but  rather  categories  of  decision  factors.  The  sub-issues 
within  a  parent  issue  constitute  the  factors.  As  discussed  later,  VFT  calls  for  factors  to  be 
weighted  so  that  the  weights  are  additive  to  1  within  each  parent  issue.  The  parent  issue 
weights  are  then  additive  to  1.  The  weighting  process  is  discussed  at  the  end  of  the  Phase 
III  section. 

Next  came  participant  evaluation  of  the  definitions  and  proposed  ranges  of  real 
values  for  each  of  the  extrinsic  factors.  Elicitation  of  the  X-axis  ranges  from  experts, 
such  as  0-100  percent  for  mission  capability  of  assets  or  0-3000  miles  for  the  range  of 
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fighter  aircraft  (refueled),  was  vitally  important  to  the  process.  Only  with  realistic  value 
ranges,  having  the  correct  upper  and  lower  bounds,  could  realistic  value  transformations 
occur. 

Once  the  factor  definitions  and  ranges — along  with  one  or  both  of  the 
hierarchies — were  deemed  correct,  the  painstaking  process  of  single-dimension  value 
function  elicitation  could  begin. 

Each  extrinsic  factor’s  blank  elicitation  graph  was  presented  to  the  experts  one  at 
a  time.  Each  expert  was  asked  to  identify  as  closely  as  possible  the  point  along  the  X- 
axis  (the  factor’s  real  range)  that  his  or  her  transformed  value  began  to  drop  from  its  high 
of  1.  The  expert  was  then  asked  to  identify  at  which  point  the  transformed  value  began  to 
increase  from  its  low  of  0.  Both  points  were  marked  in  pencil  directly  on  the  particular 
graph.  These  values  did  not  necessarily  coincide  with  the  extreme  points  of  the  factor’s 
pre-determined  real  range. 

Once  the  extreme  points  of  zero  value  and  full  value  were  identified,  the  expert 
was  asked  to  identify  the  transformed  (Y-axis)  value  that  the  values  within  the  real  range 
represented  for  him  or  her.  This  was  accomplished  via  the  drawing  or  sketching  of  a  line 
or  curve  from  one  extreme  to  the  other.  This  line  could  take  any  form  the  decision  maker 
deemed  appropriate.  The  process  was  repeated  for  every  extrinsic  factor  with  the 
exception  of  all  go/no  go  constraints,  for  which  transformed  values  are  necessarily  1  or  0. 

Once  all  of  the  graphical  representations  of  value  had  been  elicited  from  all  of  the 
participating  experts,  that  portion  of  the  interview  sessions  was  complete.  Mathematical 
transformation  of  each  curve  or  line  into  a  single-dimension  value  function  was  held  for 
later,  upon  completion  of  all  interviews.  This  was  done  using  Microsoft  Excel 
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spreadsheet  applications  to  create  linear,  piecewise  linear,  and  exponential  functions  that 
would  be  used  to  convert  candidate  force  mix  scores  to  values  between  1  and  0  for  each 
factor. 

Upon  completion  of  the  last  graphical  factor  value  representation,  each  participant 
was  asked  to  provide  a  rank  ordering  for  all  26  extrinsic  factors.  These  rank  ordering 
sheets,  an  example  of  which  can  be  found  in  Appendix  E,  provided  a  grouping  according 
to  relative  importance  that  enhanced  the  development  of  the  factor  weighting 
questionnaires  used  in  the  final  rounds  of  modified  Delphi,  the  last  step  of  Phase  III. 

However,  the  in-person  interviews  did  not  end  with  elicitation  of  the  relative 
rankings.  At  the  end  of  each  interview,  every  participant  was  asked  his  or  her  opinion  on 
the  relationship  between  extrinsic  and  intrinsic  factors;  such  as  which,  if  any,  should  be 
weighted  more  than  the  other.  Particularly,  participants  were  asked,  “If  the  extrinsic 
value  of  candidate  mix  A  were  .7  and  the  intrinsic  value  were  .3,  how  would  that  rate 
against  mix  B,  of  which  the  sum  is  equal  but  the  values  are  reversed?”  These  discussions 
were  almost  completely  unstructured  and  sometimes  lengthy.  This  was  desirable  because 
the  intent  was  to  develop  a  theory  of  the  relationship  that  could  be  accurately  modeled 
mathematically  and  would  subsequently  answer  research  question  6. 

The  final  step  of  Phase  III  was  the  final  round  of  modified  Delphi  study — 
administering  of  the  questionnaires  that  developed  the  relative  weights  for  the  factors. 
Each  participant  in  this  phase  was  first  emailed  an  instructional  talking  paper  on  the 
characteristics  of  performing  swing  weight  analyses.  This  was  done  three  days  prior  to 
the  emailing  of  the  actual  questionnaires  to  ensure  that  each  participant  thoroughly 
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understood  the  process.  The  talking  paper  was  a  modified  version  of  that  used  by  Buzo 
[Buzo,  2000:  174-176],  and  can  be  viewed  in  Appendix  F. 

Having  given  the  participants  three  days  to  digest  the  process,  the  modified 
Delphi  round  two  questionnaires  were  emailed.  The  rank  ordering  of  factors 
accomplished  earlier  during  the  in-person  interviews  provided  for  the  structure  of  the 
questionnaire,  allowing  for  the  order  by  which  sub-issues  (the  factors)  would  be  weighted 
against  each  other.  Hence,  the  highest-ranking  sub-issue  within  each  parent  issue  could 
be  compared  to  every  other  sub-issue  within  the  same  parent  issue. 

Within  the  questionnaire,  all  of  the  necessary  comparisons  were  put  in  table  form. 
A  narrative  explained  the  process  and  provided  yet  another  swing  weight  example  to 
ensure  participant  understanding.  The  questionnaire  elicited  only  one  task,  worded  as 
follows:  For  each  of  the  factors  or  parent  issues  presented,  “ state  how  many  times  more 
important  the  swing  from  best  score  to  worst  score  for  the  issue  in  Column  A  is  than  the 
swing  from  best  score  to  worst  score  for  the  issue  in  Column  B  ”  [Buzo,  2000:  59].  This 
wording  from  Buzo’s  work  was  left  completely  as  is,  to  ensure  reasonable  duplication  of 
his  weighting  elicitation  process.  The  questionnaire  also  asked  for  any  comments  the 
participant  might  want  to  provide  as  justification  for  his  or  her  weightings. 

Swing  weight  analysis  calls  for  the  comparison  between  two  factors’  loss  of  value 
along  their  entire  ranges,  from  highest  possible  scores  to  lowest  possible  scores.  The 
decision  maker  then  need  only  determine  which  loss  of  value  is  more  important  to  him  or 
her  and  by  how  much  (or  how  many  times).  Analyzing  the  swing  in  value,  the  decision 
maker  might  determine  that  A’s  complete  loss  in  value  is  twice  as  important  as  B’s. 
Therefore,  A  would  have  twice  the  weight  of  B,  or  by  equation,  A  =  2B  . 
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Within  its  response  table,  the  questionnaire  also  provided  for  such  comparisons 
between  parent  issues,  the  ordering  of  which  was  also  determined  by  the  earlier  in-person 
rank  ordering  of  all  factors.  Those  factors  that  had  been  ranked  higher  by  the  round  1 
participants  implied  that  their  parent  issues  were  of  greater  importance.  Of  course,  this 
was  not  accepted  as  fact  but  stood  only  for  the  development  of  the  questionnaire  in  order 
to  provide  for  the  comparisons.  The  questionnaire  used  for  the  study  is  exhibited  in 
Appendix  G. 

Once  all  of  the  questionnaires  were  completed  and  returned,  each  factor’s  highest 
and  lowest  weights,  along  with  their  respective  average  weights,  were  tabulated.  The 
same  was  done  for  parent  issues.  The  summarized  results,  along  with  the  anonymous 
comments  of  the  participants,  were  attached  to  a  second  questionnaire  and  emailed  to  the 
participants.  The  omission  of  swing  weight  process  instructions  notwithstanding,  this 
second  questionnaire  was  a  duplicate  of  the  first.  Participants  were  asked  to  evaluate  the 
intermediate  results  and  anonymous  comments,  re-evaluate  their  own  judgements,  and 
return  their  updated  questionnaires.  The  results  of  this  third  and  final  round  of  modified 
Delphi  were  then  averaged  for  every  factor  and  parent  issue,  and  these  were  normalized 
to  sum  to  1  within  parent  issues  and  again  among  parent  issues. 

Phase  IV:  Development  of  the  Decision  Support  Tool 

Phases  I-III  of  research  provided  a  step-by-step  approach  to:  First,  gathering  the 
necessary  subjective  information;  then,  choosing  the  most  appropriate  DA  methodology 
for  modeling  and  constructing  the  decision  support  tool;  and  finally,  converting 
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subjective  inputs  into  quantitative  values  and  relative  weights  with  which  to  formulate  an 
objective  value  function  for  scoring  any  candidate  force  mix. 

The  two  objectives  for  Phase  IV  were  to  develop  the  final  decision  support  tool 
and  to  develop  a  mathematical  formula  to  represent  the  relationship  between  this  tool  and 
the  intrinsic  value  of  a  force  mix. 

To  develop  the  VFT  decision  support  tool,  all  of  the  individual  graphical 
representations  of  value  for  each  of  the  factors  had  to  be  converted  into  single-dimension 
value  functions.  Additionally,  the  factor  weights  had  to  be  converted  into  global  weights 
to  represent  each  factor’s  importance  within  the  overall  model.  More  on  global  weights 
is  presented  later. 

To  convert  raw  graphical  representations  into  single-dimension  value  functions, 
Kirkwood  provides  piecewise  linear  and  exponential  function  formulations  (written  in 
Visual  Basic)  for  Microsoft  Excel  spreadsheets  [Kirkwood,  1997;  62-68]  of  which  both 
Buzo  and  this  researcher  made  extensive  use  [Buzo,  2000:  24].  These  formulas  provide 
for  relatively  simple  inverse  transformation  of  a  decision  maker’s  line  or  curve  into  a 
corresponding  value  between  0  and  1 .  Creating  single-dimension  value  functions, 
however,  is  just  the  beginning. 

Each  value  function  must  have  the  appropriate  level  of  importance,  or  weight, 
within  the  model.  This  then  becomes  its  global  weight.  Phase  Ill’s  modified  Delphi 
process  provided  the  foundation  for  this  with  its  comparisons  between  factors. 

To  begin,  all  of  the  weights  for  the  sub-factors  within  a  parent  issue  must  sum  to  1 .  For 
example,  if  X  and  Y  are  the  only  two  factors  within  parent  issue  P,  and  X  =  2Y  ,  then  the 
weight  for  X  is  .67  and  the  weight  for  Y  is  .33  (rounded  here).  But  these  are  only  relative 
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weights  within  the  parent  issue.  To  become  global  weights  that  reflect  each  factor’s 
importance  to  the  model,  each  of  the  factor  weights  must  be  multiplied  by  the  weight  of 
the  parent  issue  (which  is  relative  to  the  other  parent  issues  and  of  which  all  must  also 
sum  to  1).  If  the  weight  of  P  is  .4,  then  the  global  weights  for  X  and  Y  are  .27  and  .13 
(rounded),  respectfully.  Global  weights  were  calculated  for  all  of  the  factors. 

Because  the  goal  of  this  research  is  to  create  a  decision  support  tool  with  USAF- 
wide  applicability,  single-dimension  value  functions  had  been  elicited  from  multiple 
groups  of  experts.  Buzo  had  elicited  just  one  value  function  for  each  factor  [Buzo,  2000]. 
This  research  elicited  four  separate  and  distinct  functions  for  each.  Although  simple 
averaging  of  these  functions  is  advocated  by  some  [Miller,  1970;  Joshi,  1980],  specific 
guidance  on  how  to  do  so  was  not  provided,  and  such  should  not  be  done  in  this  case 
without  justification.  Given  that  averaging  of  value  functions  could  be  accomplished, 
sensitivity  analysis  on  each  factor’s  impact  on  the  rankings  of  candidate  force  mixes 
would  either  provide  justification  for  averaging  the  value  functions  or  necessitate  a 
feedback  loop  with  the  experts  to  modify  those  functions.  However,  before  sensitivity 
analysis  could  be  accomplished,  a  set  of  candidate  force  mixes  had  to  be  found. 

A  straw  model  of  candidate  force  mixes  was  developed  based  on  a  notional  level 
of  capability  to  be  met,  matched  to  notional  suitability  ratings  for  individual  aircraft. 

Five  aircraft  types,  three  fighters  and  two  bombers,  were  included  in  the  model.  They 
were  each  given  notional  suitability  ratings  for  each  of  three  mock  aerospace  missions 
(AA,  AG,  PB).  These  aircraft  types  were  then  combined  into  51  distinct  force  mixes — all 
sufficient  to  attain  a  basic  capability  of  10  “AA”,  20  “AG”,  and  2  “PB”  aerospace 
missions,  respectively.  This  model  provides  for  multiple  force  mix  combinations  that 
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each  present  an  intrinsic  value  to  accomplish  specified  aerospace  missions.  These 
combinations  can  then  be  evaluated  within  the  VFT  extrinsic  factor  decision  support  tool 
to  generate  an  overall  extrinsic  value.  With  such  a  straw  model,  sensitivity  analyses  of 
the  individual  single-dimension  value  functions  could  progress. 

For  the  sensitivity  analysis,  the  value  functions  that  Buzo  had  elicited  from 
CentAF  were  considered  valid  groundwork  to  be  included  in  the  averaging  of  the  newly 
attained  value  functions  of  this  research.  Therefore,  each  factor  had  five  single¬ 
dimension  value  functions.  These  value  functions  were  averaged  for  each  factor,  and  the 
model  assembled  using  these  aggregated  values.  Using  the  developed  straw  model, 
candidate  force  mixes  were  entered.  This  produced  scores  that  resulted  in  a  rank  ordering 
of  force  mixes  from  most  desirable  to  least  desirable. 

Using  this  rank  order  of  candidate  force  mixes  as  the  baseline,  the  single¬ 
dimension  value  function  of  one  factor  at  a  time  was  altered  to  represent  a  single 
agency’s  value  function  (for  example,  AEFC’s  value  function  for  Resupply) — and  the 
model  was  run  again.  In  each  case,  the  single  agency  value  function  that  was  most 
dissimilar  to  the  USAF-wide  function  was  selected.  This  produced  subsequent  sets  of 
rank-ordered  candidate  force  mixes.  The  process  was  repeated  for  all  of  the  16  extrinsic 
decision  factors. 

If  an  agency’s  single-dimension  value  function  produced  a  different  rank  order 
among  any  of  the  top  three  alternatives — with  all  of  the  remaining  averaged  value 
functions  remaining  constant — then  that  elicited  value  function  failed  the  sensitivity 
analysis  and  feedback  with  the  originating  expert  was  in  order.  If  the  agency’s  value 
function  for  a  single  factor  did  not  change  the  rank  order,  then  that  factor’s  averaged 
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function  was  deemed  an  appropriate  representation  of  the  expert  group’s  judgement.  A 
second  decision  rule  was  that,  if  a  participating  expert  was  unwilling  to  alter  his  or  her 
value  function  in  light  of  having  failed  the  sensitivity  analysis,  then  all  of  his  or  her  value 
functions  would  be  removed  from  the  model,  with  appropriate  documentation  of  the 
anonymous  dissenting  opinion.  The  underlying  reasons  for  the  dissenting  perspective 
would  then  be  investigated. 

Sensitivity  analyses  was  also  necessary  for  the  parent  weights  within  the  decision 
support  tool.  This  was  accomplished  by  altering  one  parent  weight  at  a  time,  but  only  as 
far  as  its  minimum  and  maximum  weights  as  had  been  elicited  by  the  experts.  When  a 
weight  was  altered,  the  tool  was  run  with  the  resultant  rank  ordering  of  candidate  force 
mixes  observed.  This  provided  valuable  insights  as  to  both  the  factors  and  the  weights  of 
those  factors  within  the  model.  Sensitivity  analyses  of  the  value  functions  and  parent 
weights  constituted  tests  internal  to  the  model. 

Testing  the  external  sensitivity  of  the  model  required  investigation  as  to  its  affect 
on  an  intrinsically  scored  and  ranked  set  of  force  mixes  (using  notional  suitability  scores 
attained  via  the  straw  model).  Discussion  of  this  sensitivity  analysis  is  reserved  for 
Chapter  V. 

This  fourth  phase  of  research  also  involved  an  effort  to  generate  a  formula  that 
accurately  represents  the  relationship  between  the  extrinsic  value  and  intrinsic  value  of 
force  mixes.  Overall  force  mix  value  would  be  a  combination  of  both.  While  research 
into  the  intrinsic  value  of  force  mixes  is  beyond  the  scope  of  this  effort,  it  stands  to 
reason  that  a  quantified  intrinsic  value,  once  known,  can  be  mated  with  a  quantified 
extrinsic  value.  The  challenge  was  to  develop  a  working  theory  of  the  possible  group 
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preference  of  one  over  the  other  as  discussed  earlier.  If  no  preference  exists,  simply 
summing  or  multiplying  the  two  values  could  produce  a  sound  result.  However,  if  global 
preference  of  one  over  the  other  existed,  however  slight,  it  must  be  reflected  in  the 
mathematical  relationship  in  order  to  ensure  maximum  representation  of  the  subject 
matter  expert’s  group  judgement.  Feedback  from  the  expert  participants  showed  just 
such  global  preference  for  intrinsic  over  extrinsic. 

Phase  IV  concluded  with  an  investigation  of  how  to  mathematically  reflect  a 
slight  preference  for  intrinsic  value  over  extrinsic  value.  This  centered  around  the 
insightful  concept  that  intrinsic  value  is  a  combination  of  capability  and  suitability,  and 
that  only  one  of  the  two  required  preferential  status  in  force  mix  scoring. 

Summary 

This  chapter  presented  a  step-by-step  explanation  of  the  four  phases  of  research 
used  to  develop  an  extrinsically-oriented  decision  support  tool  to  aid  campaign  planners 
in  selecting  the  most  appropriate  force  mixes  during  crisis  action  planning.  Numerous 
objectives  were  established  to  properly  address  the  six  research  questions.  All  of  these 
objectives  were  met. 

The  Phase  I  section  discussed  the  separation  of  extrinsic  and  intrinsic  factors  and 
elicitation  of  the  definitions  for  each  category;  elicitation  and  extraction  of  the  relevant 
decision  factors;  subjective  expert  evaluation  of  the  independence  and  mutual  exclusivity 
of  each;  and  definitions  for  each. 

The  Phase  II  section  discussed  the  conduct  of  the  literature  review  to  determine 
the  most  appropriate  DA  technique  for  this  research  situation. 
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The  three-round  modified  Delphi  study  was  addressed  in  the  Phase  III  section, 
with  the  focus  on  eliciting  quantified  single-dimension  value  functions  and  determining 
relative  weights  of  factors  within  the  model.  Also  discussed  were  the  subjective  expert 
evaluations  of  candidate  hierarchies  and  the  preliminary  formulation  of  a  theory 
pertaining  to  the  relationship  between  extrinsic  value  and  intrinsic  value. 

Finally,  the  Phase  IV  section  outlined  the  decision  tool  development  including 
comprehensive  sensitivity  analyses  on  both  value  functions  and  global  weights.  It  also 
addressed  the  trial  and  error  mathematical  investigation  to  reflect  slight  preference  for 
intrinsic  force  mix  value  over  extrinsic  value. 

The  next  chapter  presents  the  results  of  the  overall  research  effort. 
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V.  Results 


Phase  I:  The  CDM  Interviews  and  Content  Analysis 

CDM  interviews  and  content  analysis  addressed  research  questions  1  and  2,  while 
the  selection  of  the  participating  subject  matter  experts  provided  the  first  step  for 
addressing  question  5. 

All  of  the  34  study  participants  agreed  on  the  existence  of  both  extrinsic  and 
intrinsic  campaign  planning  decision  factors,  and  on  the  clear  separation  between  the  two 
types.  This  unanimous  agreement  was  important  to  the  basic  premise  of  this  research — 
that  a  decision  support  model  could  focus  solely  on  extrinsic  factors  not  pertaining  to  the 
mission-specific  or  absolute  capabilities  of  candidate  force  mixes. 

Many  potential  definitions  surfaced  for  what  constitutes  an  extrinsic  factor,  and 
most  could  prove  suitable.  Definition  2  incorporates  the  relevant  inputs  from  the  subject 
matter  experts:  Extrinsic  decision  factors  are  situational  considerations,  external  to  and 
independent  of  a  weapons  platform’s  design,  that  affect  determination  of  the 
appropriateness  and  subsequent  goodness  of  that  weapons  platform  for  a  specified 
mission — given  a  specific  contextual  environment  that  requires  evaluation. 

Likewise,  there  were  many  definitions  presented  for  what  constitutes  an  intrinsic 
decision  factor,  and  the  relevant  inputs  have  been  incorporated.  Definition  1  is:  Intrinsic 
factors  represent  a  weapons  platform’s  fundamental  ability  to  accomplish  a  specified 
aerospace  mission  or  tasking,  and  are  limited  to  determinations  of  efficacy  in  a  given 
situation  related  only  to  the  designated  purpose  of  its  design. 
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In  addition  to  defining  extrinsic  and  intrinsic  factor  types,  this  phase  elicited  an 
exhaustive  list  of  22  mutually  independent,  specific  extrinsic  factors.  The  final  factors 
included  in  the  decision  support  model  can  be  viewed  in  Appendix  C,  with  their 
associated  descriptions  in  Appendix  D.  These  are  the  extrinsic  decision  factors  important 
to  campaign  planners  in  selecting  force  mixes  in  response  to  theater  crises.  The  claim  of 
an  exhaustive  list  is  based  on  the  34  interviews  with  experts  and  the  content  analysis  of 
USAF  and  Joint  guidance,  all  of  which  combined  to  produce  just  these  factors.  Tests  for 
independence  were  comprised  of  conceptual  implementations  of  Kirkwood’s  Theorems 
9.19  and  9.20  in  discussions  with  subject  matter  experts,  sans  the  graphical  and  formula 
representations  [Kirkwood,  1997:  238-239]. 

Can  the  factors  affecting  force  mix  selection  in  response  to  theater  crises  be 
clearly  categorized  as  extrinsic  and  intrinsic,  and  can  these  categories  be  clearly  defined? 
This  was  the  first  research  question,  and  it  has  been  satisfactorily  answered.  All  of  the 
subject  matter  experts  agreed  on  the  separation,  and  all  of  the  experts  helped  to  enhance 
the  development  of  Definitions  1  and  2. 

The  second  research  question  has  also  been  answered.  What  extrinsic  factors  are 
important  to  campaign  planners  when  selecting  aircraft  force  mixes  in  response  to  theater 
crises?  The  22  factors  identified  by  Phase  II  of  this  research,  and  presented  in 
Appendices  C  and  D,  are  those  that  are  important  to  campaign  planners. 

Phase  II:  Literature  Review  to  Determine  the  Best  DA  Method 

The  purpose  of  the  second  phase  of  research  was  to  address  research  question  4, 
which  involved  selection  of  the  most  appropriate  Decision  Analysis  (DA)  technique  to 
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model  the  decision  situation  at  hand.  A  search  of  DA  literature  provided  the  foundational 
information  necessary  to  properly  compare  DA  methodologies.  With  background  on 
various  techniques  and  advancements,  along  with  an  understanding  of  the  decision 
situation  at  hand,  comparisons  could  be  made  on  the  basis  of  knowledge  and  applicability 
rather  than  on  the  basis  of  convenience  or  the  specific  competencies  of  the  researcher. 

In  order  to  ascertain  the  most  appropriate  DA  technique,  research  question  4 
asked:  How  should  these  factors  and  the  relationships  between  them  be  modeled?  This 
question  has  been  satisfied  with  the  selection  of  Multi- Attribute  Utility  Theory  using 
Value  Focused  Thinking  (VFT).  The  selection  criteria  can  be  reviewed  in  Chapter  III. 

Selection  of  VFT  as  the  method  of  choice  necessitated  development  of  a  value 
hierarchy  with  which  to  model  the  relationship  among  extrinsic  decision  factors.  Such  a 
hierarchy  was  developed.  This  intermediate  hierarchy  was  introduced  to  experts  during 
Phase  III  of  the  study  (see  Figure  11),  along  with  the  hierarchy  developed  by  Buzo 
[Buzo,  2000:  67].  The  final  hierarchy  is  based  on  the  feedback  provided  by  the  subject 
matter  experts  and  is  presented  in  the  Phase  III  section  of  this  chapter. 

Phase  III:  Iterative  Interviews  (Modified  Delphi)  to  Attain  Consensus 

While  this  phase  of  research  addressed  all  of  the  six  research  questions,  its 
purpose  was  to  lay  the  groundwork  in  answering  question  3 — quantifying  the  extrinsic 
decision  factors. 

The  in-person,  first  round  of  modified  Delphi  study  established  the  single¬ 
dimension  value  functions  (SDVF’s)  for  each  of  the  16  scored  extrinsic  decision  factors 
(six  of  the  factors  are  go/no  go  constraints  that  are  not  given  SDVF’s).  Individual  sets  of 
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SDVF’s  were  elicited  from  AEFC,  CADRE,  SouthAF,  and  ACC,  for  a  total  of  64 
SDVF’s. 

Each  SDVF  was  transformed  from  its  hand-drawn  representation  into  a  Microsoft 
Excel-based  value  curve  using  “ValueE”  and  “ValuePL”  functions  [Kirkwood,  1997:  81]. 
Coding  of  these  functions  was  through  Microsoft’s  Visual  Basic  programming  and  can  be 
viewed  in  Appendix  M.  Initial  quantification  of  the  extrinsic  factors  was  complete.  This 
involved  quantification  relative  to  each  agency  independently.  The  final  quantification  of 
factors,  combining  all  into  USAF-Wide  representations,  was  reserved  for  the  final  phase 
of  research. 

Phase  III  also  allowed  for  expert  feedback  on  the  two  candidate  VFT  hierarchies, 
which  was  crucial  to  the  establishment  of  parent  issues  and  the  development  of  a  go/no 
go  filter  for  the  hierarchy.  Of  the  eight  experts  who  claimed  preference  for  one  hierarchy 
over  the  other,  six  selected  the  version  established  by  this  research.  Aside  from  their 
assertions  that  it  presents  a  better  conceptual  representation  of  the  decision  situation,  all 
six  experts  showed  preference  for  a  model  that  separates  feasible  force  mixes  from 
infeasible  mixes  prior  to  the  scoring.  Such  separation  allows  decision  makers  to  accept  a 
group  of  feasible  mixes  and  bypass  the  extrinsic  scoring  function  of  the  decision  support 
tool  if  they  so  choose.  The  final  value  hierarchy  representing  this  decision  situation  is 
presented  in  Figure  13. 
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Figure  13:  Final  VFT  value  hierarchy 


As  Figure  13  shows,  the  four  parent  issues  are  Host  Nation/Base,  Operating 
Location,  Force  Mix  Usage,  and  Enemy  Considerations.  Establishment  of  these  parent 
issues  was  important  to  the  weighting  of  individual  factors  within  the  model,  and  to  the 
sensitivity  analysis  on  the  model  weights  which  is  discussed  later.  The  go/no  go  filter 
was  placed  above  the  scored  factors  in  the  hierarchy  to  represent  its  importance  as  a 
determinant  of  which  force  mixes  will  be  scored. 

The  second  and  third  rounds  of  modified  Delphi  study  established  the  parent  issue 
weights  and  global  weights  for  the  factors  within  the  model.  These  rounds  were  in  the 
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form  of  email  questionnaires.  Instructions  for  the  questionnaires  were  provided  and  can 
be  viewed  in  Appendix  F.  A  sample  of  the  questionnaire  is  shown  in  Appendix  G. 

Elicitation  of  pairwise  comparisons  of  relative  importance  among  all  of  the  scored 
factors  generated  one  complete  set  of  weights  for  each  of  the  four  agencies  (ACC,  AEFC, 
CADRE,  SouthAF).  These  weights  were  averaged  and  re-submitted  to  the  experts,  who 
were  given  an  opportunity  to  revise  or  defend  their  previous  inputs.  The  result  is  a  set  of 
global  weights  representing  the  relative  importance  of  each  scored  factor  within  the 
model.  It  is  important  to  note  that,  just  as  the  CentAF  value  functions  (elicited  from 
previous  research)  are  included  in  the  model,  the  CentAF  global  weights  are  likewise 
included,  averaged  into  the  total. 

The  complete  set  of  global  weights  and  parent  issue  weights,  along  with  the 
lowest  and  highest  weights  elicited,  are  presented  in  Table  5. 

Research  Question  3  asked:  How  are  these  factors  quantified,  and  what 
are  the  relationships  between  them?  This  research  question  has  been  satisfied.  Extrinsic 
factors  are  best  quantified  by  placement  into  a  VFT  value  hierarchy,  with  the  assignment 
of  individual  single-dimension  value  functions  (SDVF’s)  and  global  weights  to  each. 

The  hierarchy  is  presented  in  Figure  13,  and  the  global  weights  are  provided  in  Table  5. 
However,  individual  agency  SDVF’s,  while  specific  quantifications  of  each  factor,  are 
not  considered  the  final  quantifications  of  each.  It  is  the  USAF-wide  SDVF’s  discussed 
in  the  next  section  that  are  the  driving  force  behind  the  model. 
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Table  5:  VFT  model  weights 


AVERAGE  WEIGHTS 

LOW 

AVG 

HIGH 

ROUNDED 

MN  Compatibility 

0.018786 

0.049763 

0.090226 

0.049 

Ability  to  Resupply 

0.00738 

0.049823 

0.174 

0.048 

POL  Storage 

0.067007 

0.080772 

0.098626 

0.08 

Fighter  Distance 

0.019145 

0.030507 

0.049313 

0.03 

Bomber  Distance 

0.006382 

0.026225 

0.03 

0.026 

MSA:  Fighter 

0.024657 

0.06328 

0.179999 

0.063 

MSA:  Bomber 

0.012328 

0.056241 

0.179999 

0.056 

Climate 

0.003522 

0.018234 

0.03 

0.018 

Topography 

0.015 

0.031175 

0.067 

0.031 

Tanker  Support 

0.022336 

0.062375 

0.090226 

0.062 

Force  Mix  Flexibility 

0.011272 

0.036324 

0.067 

0.035 

Force  Mix  Utilization 

0.011272 

0.031337 

0.051 

0.03 

MC  Rate 

0.045 

0.072559 

0.107211 

0.072 

Enemy  vs  USAF  Capability 

0.09 

0.210197 

0.394505 

0.209 

Enemy  ISR 

0.03 

0.095283 

0.178685 

0.095 

Enemy  Interference 

0.09 

0.096773 

0.134014 

0.096 

PARENT  ISSUES,  AVG  WTS 

_  LOW 

AVG 

HIGH 

1 

Host  Nation 

0.025831 

0.097 

0.120301 

Basing 

0.215061 

0.366 

0.625002 

Usage 

0.078901 

0.137 

0.180451 

Enemy 

0.209999 

0.4 

1 

0.624633 

Phase  IV:  Development  of  the  Decision  Support  Tool 

Phase  I  confirmed  the  separation  between  extrinsic  and  intrinsic  factor  types  and 
developed  their  definitions.  It  also  established  which  specific  extrinsic  factors  are 
important.  Phase  II  established  the  most  appropriate  method  for  modeling  the  problem. 
Phase  III  confirmed  the  final  value  hierarchy,  final  parent  issue  weights  and  global 
weights,  and  the  agency-specific  SDVF’s  for  each  of  the  scored  factors.  Phase  IV  of  the 
research  integrated  all  of  the  results  into  a  viable  decision  support  model. 
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The  primary  purpose  of  Phase  IV  was  two-fold:  to  develop  an  actual  decision 
support  tool  that  provides  USAF-wide  applicability  (research  question  5),  and  to  provide 
an  accurate  mathematical  representation  of  the  relationship  between  intrinsic  value  and 
extrinsic  value  (research  question  6). 

The  developed  single-dimension  value  functions  (SDVF’s)  are  the  heart  of  the 
VFT  decision  support  tool.  It  is  these  values  that  reflect  the  subjective  judgements  of  the 
participating  experts  in  a  quantitative  fashion,  allowing  for  quantification  of  the  entire 
decision  situation. 

Single  dimension  value  functions  were  elicited  from  AEFC,  ACC,  CADRE, 
SouthAF,  and  CentAF  (the  last  via  prior  research).  This  produced  five  SDVF’s  for  each 
extrinsic  factor — a  contribution  of  80  for  the  model.  Note  that  the  previous  research  had 
established  only  16  such  SDVF’s  [Buzo,  2000].  Successfully  combining  these  80 
SDVF’s  (five  per  factor)  into  16  (one  per  factor)  was  important  to  establishing  USAF- 
wide  applicability  for  the  model. 

For  each  factor,  the  individual  SDVF’s  were  analyzed  to  determine  which 
involved  the  highest  number  of  functions  in  its  makeup.  For  example,  one  SDVF  might 
have  a  piecewise  linear  function  for  the  first  third  of  its  range,  then  an  exponential 
function  for  the  next  third,  and  finally  either  full  (1)  or  zero  value  for  the  final  third,  with 
no  function.  If  this  SDVF  had  more  of  these  components  than  any  of  the  others  (for  this 
factor)  then  the  final  USAF-wide  SDVF  must  possess  at  least  that  many  components. 
Once  the  number  of  required  components  had  been  established,  average  midpoints  among 
all  five  SDVF’s — for  each  of  the  component  ranges — were  determined.  Finally,  the 
appropriate  functions,  either  exponential  or  piecewise  linear,  were  input  for  each 
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component  range,  producing  an  overall  USAF-wide  SDVF  that  accurately  represents  the 
preferences  of  the  decision  makers.  This  was  repeated  for  all  16  factors.  That  done,  the 
model  contains  a  total  of  96  SDVF’s.  The  16  USAF-Wide  SDVF’s  produced  by  this 
process  can  be  used  in  computing  the  extrinsic  values  of  force  mixes — with  USAF-wide 
applicability.  However,  any  of  the  original  sets  of  agency-specific  SDVF’s  can  also 
compute  the  extrinsic  value  of  a  candidate  force  mix. 

Research  questions  3  and  5  have  been  satisfied.  How  are  these  factors  quantified 
and  what  are  the  relationships  between  them?  They  are  quantified  via  single-dimension 
value  functions  within  an  additively-independent  VFT  value  hierarchy  with  global 
weights  determining  relative  importance.  Can  the  research  results  be  applied  USAF- 
wide?  The  extrinsic  factors  have  been  quantified  with  USAF-wide  applicability  through 
the  averaging  of  their  SDVF’s  (and  the  keeping  of  all  single  agency  SDVF  sets  within  the 
model),  and  the  global  weights  have  USAF-wide  applicability  because  the  three-round 
modified  Delphi  study  produced  weights  that  proved  robust  during  sensitivity  analyses. 

Each  of  the  USAF-wide  single-dimension  value  functions,  along  with  the  values 
for  the  components  of  the  individual  SDVF’s  that  generated  them,  can  be  seen  in 
Appendix  H.  An  important  consideration  is  that  all  of  the  96  SDVF’s  and  all  of  the 
single-agency  parent  issue  and  global  weights  remain  within  the  model.  This  gives  future 
analysts  or  decision  makers  an  opportunity  to  choose  the  group  of  experts  whose  SDVF’s 
and  weights  they  might  prefer,  if  they  disagree  with  the  USAF-wide  composition  or  if 
they  just  want  a  more  theater-specific  slant  on  decision  making  (such  as  CentAF  SDVF’s 
for  a  Southwest  Asia  engagement).  This  establishes  a  family  of  VFT  models  in  one, 
further  enhancing  the  claims  of  applicability  throughout  the  entire  Air  Force. 
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The  entire  VFT  decision  support  tool  is  in  Microsoft  Excel  spreadsheet  form.  The 
“Scored  Forces”  sheet  contains  the  formulas  for  translating  straw  model,  SDVF,  and 
constraint  (go/no  go  filter)  inputs  into  overall  force  mix  extrinsic  scores.  Each  SDVF  is  a 
combination  of  IF  statements  incorporating  all  of  its  value  curve  components.  The 
“Scored  Forces”  sheet  multiplies  the  SDVF’s  by  their  respective  go/no  go  filter  scores 
and  their  global  weights  for  all  16  factors,  for  all  the  candidate  force  mixes.  The 
individual  Excel  formulas  for  each  of  the  16  extrinsic  decision  factors  can  be  seen  in 
Appendix  N. 

Linked  to  both  the  “Scored  Forces”  sheet  and  the  “Straw  Model”  sheet  is  the 
“Constraint  Filter”  sheet  (the  go  / no  go  filter).  This  filter  improves  upon  previous 
research  in  that  it  calculates  the  model  constraints  based  upon  each  aircraft  type  within  a 
force  mix,  rather  than  just  on  the  force  mix  as  a  whole.  Therefore,  if  one  aircraft  type  is 
infeasible  (i.e.,  not  allowed  for  political  reasons),  then  the  entire  force  mix  is  infeasible. 
This  does  not  omit  from  consideration  the  otherwise  qualified  aircraft  within  the  force 
mix  because  they  are  scored  as  parts  of  other  candidate  force  mixes.  Another 
improvement  is  that  some  of  the  constraints,  such  as  unrefueled  distance  from  targets,  are 
linked  to  and  derived  from  the  straw  model. 

The  “Straw  Model”  sheet  is  a  necessary  component  of  the  decision  model  because 
it  provides  the  avenue  for  testing.  The  straw  model  developed  for  this  research 
incorporates  a  matrix  of  five  weapons  platform  types  and  three  required  aerospace 
missions.  Capability  can  be  set  as  a  requirement  and  combinations  of  forces  that  meet  the 
requirements  can  be  tabled  and  their  intrinsic  suitability  computed  (with  notional  data). 
This  construction  parallels  the  makeup  of  concurrent  AFIT  research  into  the  scoring  of 
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intrinsic  value  and  allows  for  continuity  of  future  ALP  research.  Also,  its  format  is 
appropriate  for  the  introduction  of  real  world,  campaign-specific  data  by  decision  makers. 
The  previous  straw  model  was  comprised  essentially  of  a  single,  escalating  force  mix, 
with  random  individual  scores  for  each  force  mix  as  inputs  into  the  extrinsic  scoring 
sheet. 

Another  advantage  of  the  straw  model  is  that  the  candidate  operating  location 
determines  many  of  the  values.  This  set-up  allows  for  more  than  just  greater  realism,  it 
allows  for  decision  makers  to  score  potential  operating  locations  against  force  mixes. 
This  was  not  a  goal  of  the  research,  but  goes  far  in  enhancing  ALP  and  M-R  VAT 
development.  With  the  ALP  capability  of  generating  real-time  logistics  information  for 
all  operating  locations  worldwide,  any  such  location  could  be  instantaneously  scored  by 
entering  its  information  into  the  model. 

The  model  also  contains  individual  sheets  for  each  of  the  16  extrinsic  decision 
factors.  Graphs  are  provided  for  all  individual  agency  SDVF’s  as  well  as  each  USAF- 
wide  SDVF.  Again,  Appendix  H  shows  the  graphical  representations  for  the  USAF-wide 
functions.  The  model  contains  a  number  of  additional  sheets  pertaining  to  the  hierarchy 
and  various  sensitivity  analyses,  but  only  one  more  is  mentioned  here. 

The  parent  issue  weights  and  global  weights  are  calculated  on  the  “Computing 
Weights”  sheet,  which  is  linked  by  formula  to  the  “Scored  Forces”  sheet.  One  can  alter 
the  weight  of  any  parent  issue  and  the  others  will  adjust  automatically  in  proportion  to 
each  other.  Any  change  in  the  parent  issue  weights  is  immediately  reflected  in  the 
“Scored  Forces”  sheet,  as  is  any  change  in  the  straw  model  inputs,  operating  location 
inputs,  USAF-wide  SDVF’s,  or  the  constraints.  This  complete  linkage  makes  this 
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decision  support  tool  ready  now  for  graphical  user  interface  development.  Moreover,  the 
tool  is  ready  now  for  input  of  real-world  data  on  operating  locations  and  candidate  force 
mixes  that  will  instantaneously  generate  real-world  extrinsic  value  of  those  force  mixes. 

Three  sample  sheets  from  within  the  VFT  decision  support  tool  are  presented, 
without  formulas  due  to  space  limitations,  in  Appendix  K. 

The  sixth  and  final  research  question  called  for  an  accurate  mathematical 
representation  of  the  relationship  between  extrinsic  value  and  intrinsic  value.  While  this 
research  effort  focused  on  determining  the  extrinsic  value  of  force  mixes  only,  parallel 
AFIT  research  was  being  conducted  on  the  intrinsic  factors  [Wakefield,  2001].  However, 
it  was  the  responsibility  of  this  research  to  attempt  an  integration  of  the  two  types  of 
value.  That  said,  three  related  measures  of  value  apply  to  the  overall  effort:  capability, 
suitability,  and  extrinsic  value. 

Capability  is  defined  under  this  research  as  the  number  of  sorties  required  within 
each  aerospace  mission  type  that  is  required  by  the  joint  or  theater  commander  at  a 
specific  point  in  time.  For  example,  on  day  six  of  a  particular  campaign,  the  commander 
may  require  10  air  interdiction  sorties,  20  air-to-ground  sorties,  5  strategic  bombing 
sorties,  and  1  combat  search  and  rescue  sortie.  This  is  the  capability  requirement. 
Suitability  then  measures  each  aircraft  type ’s  relative  ability  to  accomplish  the  required 
aerospace  missions.  For  example.  Fighter  X  may  have  suitability  scores  of  .8  for  air 
interdiction,  .5  for  air-to-ground,  and  .1  each  for  strategic  bombing  and  combat  search 
and  rescue.  These  numbers  are  notional  because  research  into  determination  of  these 
actual  values  is  beyond  the  scope  of  this  research. 
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Capability  and  suitability  are  both  considered  independent  dimensions  of  intrinsic 
value.  The  remaining  research  question  (question  6)  required  investigation  of  how  all 
three  dimensions  of  value  can  be  combined  into  a  single  analytical  approach. 

In-person  interviews  had  determined  that  participating  experts  preferred,  slightly, 
intrinsic  value  to  extrinsic  value.  With  the  development  of  the  straw  model  (under  the 
premise  of  intrinsic  value  resulting  from  suitability  and  capability),  preference  for 
intrinsic  value  is  straightforward — force  mixes  that  do  not  first  meet  the  capability 
requirements  are  not  scored.  Using  attainment  of  required  capability  as  a  qualifier,  the 
task  remaining  was  to  address  how  to  best  combine  a  weapons  platform’s  suitability  with 
its  extrinsic  (situational)  value. 

As  a  notional  starting  point  for  this  investigation,  it  was  decided  to  follow  up  on 
the  relationship  initially  proposed  as  a  default  approach  by  Buzo,  a  straight  multiplicative 
relationship  [Buzo,  2000]  where  the  intrinsic  score  is  just  multiplied  by  the  extrinsic 
value  (as  determined  by  the  VFT  model). 

A  multiplicative  relationship  is  superior  to  an  additive  relationship — in  this 
situation — for  two  reasons.  First,  suitability  scores  and  extrinsic  values  will  most 
probably  be  in  different  units  of  measurement.  Second,  the  go /no  go  filter  of  the  extrinsic 
model  disqualifies  some  force  mixes  from  consideration,  giving  those  mixes  overall 
scores  of  zero.  An  additive  relationship  could  result  in  disqualified  force  mixes  holding 
considerable  value  when  combined  with  the  suitability  score,  which  would  be 
undesirable.  A  multiplicative  relationship  yields  an  overall,  combined  score  of  zero  for 
those  disqualified  mixes. 
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The  utility  of  this  proposed  multiplicative  relationship  was  tested  during  the 
sensitivity  analysis  (internal  and  external)  discussed  next.  Research  question  6, 
concerning  the  intrinsic/extrinsic  relationship,  will  be  revisited  after  presentation  and 
discussion  of  the  sensitivity  analysis  results. 

The  following  section  discusses  internal  sensitivity  analysis  (on  the  global 
weights  and  the  averaged,  US AF- wide  SDVF’s),  and  external  sensitivity  analysis  (on  the 
impact  of  the  VFT  model  on  intrinsically  scored  and  ranked  force  packages). 

Internal  Sensitivity  Analysis 

The  first  area  of  internal  sensitivity  analysis  was  the  testing  of  the  impact  that  a 
single  agency  SDVF  for  a  single  factor  would  have  on  the  results  of  the  VFT  decision 
support  tool.  The  purpose  was  to  validate  the  averaged  USAF-wide  SDVF’s  as  accurate 
representations  of  the  group’s  point  of  view. 

The  decision  support  model  was  run  16  times,  once  each  for  every  scored 
extrinsic  factor.  For  each  running,  a  different  factor  was  selected,  with  the  most 
dissimilar  single  agency  SDVF  replacing  the  USAF-wide  SDVF.  Each  running  of  the 
model  produced  its  own  ranking  of  force  mixes.  The  top  four  ranked  force  mixes  from 
the  initial  model  results  (with  all  of  the  USAF-wide  SDVF’s)  were  tracked  throughout 
the  process. 

The  results  of  this  sensitivity  analysis  validated  the  use  of  USAF-wide  SDVF’s. 
None  of  the  top  four  ranked  force  mixes  changed  their  rankings  for  any  of  the  16  model 
runs.  Graphical  representation  of  the  sensitivity  analysis,  with  narrative,  can  be  found  in 
Appendix  J.  Extending  the  analysis  to  all  of  the  ranked  force  mixes,  there  were  512 
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opportunities  for  force  mixes  to  change  their  rankings  (32  mixes  scored  16  times  each). 
Of  the  512  chances,  only  four  changes  in  rank  occurred.  These  changes  were  among 
force  mixes  that  were  quite  similar  in  basic  makeup — almost  identical.  Therefore,  the 
conclusion  is  that  the  USAF-wide  SDVF’s  are  robust  representations  of  the  values  of  the 
individual  agencies  involved. 

The  second  internal  sensitivity  analysis  had  the  purpose  of  validating  the  parent 
issue  weights  within  the  decision  support  tool.  Each  parent  issue  weight  was  changed, 
one  at  a  time,  incrementally  between  a  value  considerably  lower  than  the  lowest  weight 
received  by  any  expert  and  a  value  considerably  higher  than  the  highest  weight  received. 
With  each  change  in  a  particular  parent  issue  weight,  the  other  three  weights  were 
formulated  to  change  in  proportion  to  each  other,  keeping  the  sum  of  all  weights  equal  to 
1 .  This  effort  mirrored  the  sensitivity  analysis  conducted  in  prior  research  [Buzo,  2000], 
whereby  the  top  three  ranked  force  mixes  were  tracked  throughout  the  process. 

Within  the  range  of  weights  received,  the  parent  issue  Operating  Location  had  just 
one  change  in  the  most  preferred  among  the  top  three  ranked  force  mixes.  This  change  in 
rank,  force  mix  #18  overtaking  force  mix  #19  as  the  most  preferred,  occurred  very  near 
the  value  of  the  final  weight  used  within  the  final  model  (.366). 

The  Host  Nation  parent  issue  showed  no  change  in  ranks  among  the  top  three 
force  mixes  within  the  range  of  weights  received.  This  also  occurred  for  the  other  two 
parent  issues,  Force  Mix  Usage  and  Enemy  Considerations. 

Changes  in  ranks  did  occur  among  some  force  mixes  that  had  been  ranked  lower, 
and  among  the  top  three  force  mixes  outside  the  range  of  weights  received.  This  lends 
evidence  as  to  the  robustness  of  the  averaged  weights  used  for  the  model,  as 
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representations  of  the  expert  participants’  group  judgements.  It  also  lends  support  to  the 
position  that  the  weights  are  not  unimportant  if  taken  to  extreme  ranges  outside  the  range 
of  weights  received  or  if  the  force  mixes  are  otherwise  valued  similarly. 

The  four  graphical  representations  of  this  sensitivity  analysis  are  presented  in 
Appendix  I. 

In  summary,  the  internal  sensitivity  analysis  showed  both  the  averaged  parent 
issue  weights  and  the  averaged  USAF-wide  SDVF’s  to  be  satisfactorily  robust.  This  is 
reassuring,  particularly  for  the  SDVF’s,  because  the  averaging  of  such  functions  could  be 
controversial.  In  this  case,  both  averaging  methods  meet  the  requirements  of  the  decision 
situation. 

External  Sensitivity  Analysis 

The  external  sensitivity  analysis  had  the  purpose  of  determining  whether  the  VFT 
decision  support  tool  has  practical  significance  for  the  ALP  and  the  M-R  VAT. 
Specifically,  this  sensitivity  analysis  set  out  to  determine  the  impact  that  extrinsic  scoring 
provided  by  the  VFT  model  has  on  an  intrinsically  scored  and  ranked  set  of  force  mixes, 
using  the  multiplicative  relationship  proposed  earlier. 

In  line  with  the  premise  that  only  those  force  mixes  that  meet  basic  capability 
requirements  are  scored  intrinsically  (suitability  scores),  the  straw  model  was  set  up  for 
capability  requirements  of  10  AA,  20  AG,  and  2  PB  missions,  respectively.  The  five 
aircraft  types  were  then  given  various  notional  suitability  scores  for  each  mission  type. 
Fifty-one  combinations  of  force  mixes  were  tabled,  all  of  which  met  the  basic  capability 
requirements.  The  number  of  each  aircraft,  per  type,  that  is  assigned  to  fill  a  mission 
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capability  is  multiplied  by  the  suitability  score  (that  type  against  that  mission).  A 
notional  (but  reasonable)  matrix  of  suitability  scores  was  used  in  conjunction  with 
Wakefield’s  parallel  work  on  intrinsic  value  [Wakefield,  2001].  A  force  mix’s  suitability 
score  under  this  technique  is  simply  the  sum  of  each  aircraft  type’s  suitability  scores  for 
the  missions  filled.  For  example,  10  of  aircraft  FA  may  be  assigned  against  mission  AA. 
With  FA  given  a  suitability  score  of  .8,  the  score  for  this  assignment  of  forces  is  8.  All 
such  assignments  within  a  force  mix  are  summed  to  gain  a  total  intrinsic  value  for  the 
force  mix. 

With  the  straw  model  established,  force  mixes  were  scored  extrinsically  against 
the  VFT  model.  The  pre-determined  intrinsic  value  and  newly-computed  extrinsic  value 
were  then  multiplied  to  produce  an  overall  value.  Three  ranks  were  constructed: 
intrinsic,  extrinsic,  and  combined.  Once  this  baseline  was  established,  the  sensitivity 
analysis  involved  alteration  of  the  straw  model  and  the  VFT  model  to  extreme  values 
within  each,  with  subsequent  comparisons  of  impact. 

The  intrinsic  value  computed  by  the  straw  model  was  held  constant  for  the  first 
set  of  comparisons.  All  of  the  notional  extrinsic  multipliers  within  the  VFT  model  were 
set  at  extreme  differences  of  .9  and  .1  for  different  aircraft  types  in  order  to  create  a 
greater  difference  in  final  extrinsic  values  between  force  mixes.  With  greater  extrinsic 
differences  generated  among  the  force  mixes,  these  extrinsic  values  were  multiplied 
against  the  intrinsic  values,  with  the  new  rankings  recorded.  This  process  was  repeated 
for  internal  values  of  .8  and  .2,  .7  and  .3,  .6  and  .4,  and  .51  and  .49  within  the  VFT  model. 
As  the  VFT  model  produced  tighter  and  tighter  differences  between  scored  force  mixes, 
the  impact  of  each  iteration  upon  the  intrinsic  rankings  (held  constant)  could  be 
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measured.  An  excerpt  of  the  rank  results  is  presented  in  Table  6  (complete  tabular 
rankings  are  included  in  Appendix  O). 


Table  6:  External  Sensitivity  Analysis  (intrinsic  held  constant) 


Sensitivity  Analysis  on  Impact  of  VFT  Model 

with  INTRINSIC  internal  values  held  constant 


MIX 

Independent 

Intrinsic 

Rankings 

Combined  (overall) 
Rankings  with  internal 

extrinsic  at  .51  and  .49 

Combined  (overall) 
Rankings  with  internal 

extrinsic  at  .6  and  .4 

Combined  (overall) 
Rankings  with  internal 

extrinsic  at  .7  and  .3 

Combined  (overall) 
Rankings  with  internal 

extrinsic  at  .8  and  2 

Combined  (overall) 
Rankings  with  internal 

extrinsic  at  .9  and  .1 

Mix  1 

1 

1 

1 

6 

13 

18 

Mix  13 

2 

2 

2 

8 

16 

22 

Mix  16 

3 

3 

4 

10 

18 

24 

4 

4 

5 

14 

22 

29 

Mix  9 

5 

5 

9 

15 

20 

25 

Total  rank  changes  (out  of  51)  1  18 

■IBI 

|  44 

1  47 

1  51 

As  can  be  seen  in  Table  6,  as  the  range  of  values  within  the  VFT  model  approach 
their  most  extreme  differences,  the  model’s  impact  upon  intrinsically  scored  and  ranked 
force  mixes  is  profound.  Every  one  of  the  51  scored  force  mixes  changes  its  relative  rank 
when  combined  with  its  extrinsically  scored  value.  However,  even  at  the  tightest  internal 
ranges  in  value,  the  VFT  model  has  a  clear  impact,  changing  none  of  the  top  five  but 
changing  1 8  of  the  5 1 . 

The  second  set  of  comparisons  was  accomplished  with  the  extrinsic  VFT  model 
results  held  constant  and  the  intrinsic  values  adjusted  to  their  extremes.  For  the  first  test, 
the  FA  and  FC  aircraft  were  given  .9  suitability  for  AA  missions  and  the  FB  aircraft 
given  .1 .  These  scores  were  reversed  for  the  AG  mission.  The  B1  aircraft  was  given  .9 
suitability  for  the  PB  mission,  with  the  B2  aircraft  given  .  1 .  With  these  extreme  values 
entered,  the  resultant  intrinsic  scores  for  the  force  mixes  were  multiplied  by  the  extrinsic 
values  that  had  been  held  constant.  This  process  was  repeated  for  suitability  values  of  .8 
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and  .2,  .7  and  .3,  .6  and  .4,  and  .51  and  .49.  Again,  the  impact  of  each  iteration  was 
recorded.  An  excerpt  of  the  rank  results  is  presented  in  Figure  7  (complete  tabular 
rankings  are  included  in  Appendix  O). 


Table  7:  External  Sensitivity  Analysis  (extrinsic  held  constant) 


Sensitivity  Analysis  on  Impact  of  VFT  Model 

with  EXTRINSIC  internal  values  held  constant 

Independent 

Combined  (overall) 

Combined  (overall) 

Combined  (overall) 

Combined  (overall) 

Combined  (overall) 

Intrinsic 

Rankings  with  internal 

Rankings  with  internal 

Rankings  with  internal 

Rankings  with  internal 

Rankings  with  internal 

MIX 

Rankings 

intrinsic  at  .51  and  .49 

intrinsic  at  .6  and  ,4 

intrinsic  at  .7  and  .3 

intrinsic  at  .8  and  .2 

1 

1 

1 

1 

1 

5 

2 

2 

2 

2 

w 

3 

4 

4 

4 

4 

2 

3 

3 

3 

3 

Mix  21 

6 

5 

5 

5 

5 

Total  rank  changes  (out  of  51)  |  47 

|  39 

|  35 

1  35 

|  35 

£ _ 

_ _ _J 

Table  7  shows  the  VFT  model’s  impact  when  the  differences  in  intrinsic  internal 
values  are  extreme.  As  can  be  seen,  the  VFT  model  still  changes  some  of  the  pre¬ 
determined  ranks.  As  the  internal  values  within  the  straw  model  tighten,  the  impact  of 
the  VFT  model  increases  substantially.  At  the  tightest  internal  values  of  .51  and  .49,  47 
of  the  5 1  force  mixes  change  relative  ranks.  At  the  loosest  internal  values  of  .9  and  .  1 , 
the  extrinsic  multiplier  still  results  in  35  changes  in  rank. 

Sensitivity  Analysis  and  the  Combined  Value  Measure 

Given  realistic  values  within  each,  the  VFT  decision  support  tool  clearly  impacts 
the  rankings  of  intrinsically  scored  force  mixes.  Table  8  shows  intrinsic  ranks  and  scores, 
extrinsic  ranks  and  scores,  and  the  results  of  combining  the  two  multiplicatively.  This 
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table  reflects  the  impact  of  the  VFT  model  when  both  the  model  and  the  straw  model 
have  relatively  close  internal  values  (.6  and  .4  for  intrinsic,  closer  than  .51  and  .49  for 
extrinsic). 

Both  the  internal  and  external  sensitivity  analyses  provided  valuable  indicators  as 
to  the  robustness  of  the  weights  used  and  the  USAF-wide  SDVF’s,  as  well  as  the  impact 
the  VFT  decision  support  tool  can  have  on  intrinsically-scored  candidate  force  mixes. 

The  insights  gained  by  conducting  the  sensitivity  analyses  using  a  multiplicative 
process  and  a  notional,  matrix-driven  straw  model  (that  produced  notional  suitability 
scores  and  ranks)  helped  to  generate  the  most  appropriate  answer  to  research  question  6: 
How  can  the  now-quantified  extrinsic  factors  be  effectively  combined  with  the  intrinsic 
factors  to  develop  an  overall  asset  set  utility  or  value? 

The  intrinsic  rank  structure  is  linear  and  scalar,  just  the  opposite  of  the 
extrinsically-generated  scores/ranks  that  are  derived  from  this  research’s  qualitative 
concept  (VFT) — which  are  nonlinear  and  non-scalar.  The  distinction  between  intrinsic 
value  and  extrinsic  value  is  more  than  just  conceptual;  extrinsic  and  intrinsic  each 
comprise  distinct  dimensions  of  value.  In  line  with  the  dimensional  distinction,  both  the 
straw  model  of  this  research  and  the  algorithm  under  development  by  Wakefield’s 
parallel  research  [Wakefield,  2001]  produce  suitability  scores  in  units  dissimilar  to  the 
zero-to-one  extrinsic  scores  produced  by  the  VFT  model. 

The  main  point  is  that  this  research  produces  extrinsically-scored  force  packages 
but  cannot  intrinsically  score  those  packages.  Conversely,  Wakefield’s  parallel  research 
has  developed  an  algorithm  that  intrinsically  scores  force  packages  and  establishes  a  set 
of  non-dominated  (pareto  optimal)  force  mixes  [Wakefield,  2001],  The  force  mixes 
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within  Wakefield’s  pareto  optimal  set  are  all  essentially  tied,  with  no  means  provided  for 
the  decision  maker  to  select  the  most  appropriate  force  mix  from  the  set.  This  is  a 
fortuitous  situation  for  the  VFT  model  of  this  research. 

In  this  light,  the  simple  multiplicative  approach  is  the  most  appropriate  method  for 
integrating  intrinsic  value  and  extrinsic  value — given  that  a  set  of pareto  optimal  force 
mixes  have  been  designated  and  the  VFT  model  is  used  as  the  lower  echelon  tie  breaker. 
With  this  relationship,  the  progression  through  a  force  mix  selection  starts  with  the 
capability  requirements,  goes  on  to  determination  of  the  optimal  set  of  most  suitable 
force  mixes,  and  finally,  has  the  extrinsic  value  break  the  tie  among  those  pareto  optimal 
force  mixes. 

Research  question  6  is  satisfied  by  this  multiplicative,  tie-breaking  relationship 
that  lends  slight  preference  to  the  intrinsic  value  of  force  mixes,  as  desired  by  the  experts. 
Furthermore,  this  representation  generates  overall  scores  of  zero  for  those  infeasible  force 
mixes  and  overcomes  differences  in  the  units  of  measurement  (as  well  as  dimensional 
differences). 

Table  8  presents  the  results  of  a  single  external  sensitivity  analysis,  and  illustrates 
the  multiplicative  relationship  between  extrinsic  and  intrinsic  values.  Although  none  of 
the  intrinsically  scored/ranked  force  mixes  are  tied  (as  would  be  with  a  pareto  optimal 
set),  they  nonetheless  change  relative  ranking  when  multiplied  by  the  VFT  model  results. 

In  summary,  this  research  developed  a  VFT  decision  support  tool  that  uses  robust 
issue  weights  and  robust  averaged  SDVF’s.  The  tool  contains  a  family  of  models 
representing  the  expertise  of  five  agencies,  and  incorporates  a  multiplicative,  tie-breaking 
relationship  with  the  intrinsic  values  generated  by  parallel  AFIT  research. 
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Table  8:  Extrinsic  impact  on  intrinsic  value 


MIX 

(suitability) 

Extrin  @  |_ 

Intrin  @  [ 

Multiplic.  [ 

Typical 

Rank 

.6  and  .4 

Rank 

Result 

Rank 

1 

0.713534 

15 

19.2 

1 

13.69985 

1 

2 

0.714978 

8 

13.2 

44 

9.437715 

44 

3 

0.71603 

2 

12.8 

50 

9.165182 

50 

4 

0.714309 

9 

18.8 

7 

13.42902 

7 

5 

0.710151 

34 

15.2 

21 

10.7943 

23 

6 

0.70924 

38 

13.2 

44 

9.361965 

45 

7 

0.711153 

25 

14.8 

37 

10.52506 

42 

8 

0.709751 

35 

12.8 

50 

9.084811 

51 

9 

0.710476 

30 

19.2 

1 

13.64113 

2 

10 

0.712788 

16 

15.2 

21 

10.83438 

22 

11 

0.711142 

26 

18.8 

7 

13.36946 

8 

12 

0.714035 

10 

14.8 

37 

10.56772 

37 

13 

0.713743 

12 

19 

3 

13.56112 

4 

14 

0.715651 

4 

13 

46 

9.303466 

46 

15 

0.715283 

5 

13 

46 

9.298677 

47 

16 

0.71402 

11 

19 

3 

13.56638 

3 

17 

0.710439 

31 

15 

28 

10.65658 

29 

18 

0.709478 

36 

13 

46 

9.223216 

48 

19 

0.710439 

31 

15 

28 

10.65658^ 

29 

20 

0.709478 

36 

13 

46 

9.223216 

48 

21 

0.710648 

27 

19 

3 

13.50231 

5 

22 

0.713594 

13 

15 

28 

10.7039 

25 

23 

0.710648 

27 

19 

3 

13.50231 

5 

24 

0.713594 

13 

15 

28 

10.7039 

25 

25 

0.715222 

6 

15.2 

21 

10.87138 

21 

26 

0.716797 

1 

14.8 

37 

1 0.60859 

35 

27 

0.715707 

3 

15 

28 

10.7356 

24 

28 

0.711992 

23 

17.2 

9 

12.24627 

9 

29 

0.712621 

20 

16.8 

13 

11.97203 

14 

30 

0.712379 

22 

17 

11 

12.11044 

10 

31 

0.70073 

50 

15.2 

21 

10.65109 

31 

32 

0.710606 

29 

14.8 

37 

10.51697 

43 

33 

0.70368 

45 

15 

28 

10.5552 

38 

34 

0.71145 

24 

16 

17 

11.3832 

16 

35 

0.704337 

44 

16 

17 

11.2694 

19 

36 

0.708184 

40 

15 

28 

10.62276 

34 

37 

0.704945 

43 

16.2 

15 

11.42011 

15 

38 

0.701729 

49 

16.2 

15 

11.36802 

17 

39 

0.700554 

51 

15.2 

21 

10.64843 

32 

40 

0.71521 

7 

15.8 

19 

11.30033 

18 

41 

0.710439 

33 

15.8 

19 

11.22494 

20 

42 

0.712593 

21 

14.8 

37 

10.54638 

41 

43 

0.705602 

41 

15 

28 

10.58403 

36 

44 

0.70902 

39 

15 

28 

10.6353 

33 

45 

0.705602 

41 

17 

11 

11.99523 

12 

46 

0.702531 

46 

15.2 

21 

10.67847 

27 

47 

0.702531 

46 

15.2 

21 

10.67847 

27 

48 

0.702531 

46 

17.2 

9 

12.08353 

11 

49 

0.712772 

17 

14.8 

37 

10.54903 

39 

50 

0.712772 

17 

14.8 

37 

10.54903 

39 

51 

0.712772 

17 

16.8 

13 

11.97457 

13 

* 

Shading  denotes  a  rank  change 
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VI.  Conclusion 


Introduction 

Chapters  I  and  II  discussed  the  Defense  Advanced  Research  Projects  Agency’s 
Advanced  Logistics  Project  (ALP),  stating  that  the  ALP  will  provide  near  real-time 
logistics  information  from  Department  of  Defense  and  Allied  organizations  throughout 
the  world.  Campaign  planners  will  have  an  opportunity  to  compare  deployment 
scenarios  with  accurate,  up-to-date  information  and  respond  to  crises  with  greater 
efficiency  than  previously  possible. 

As  a  logistics-centered  computing  architecture,  the  ALP’s  overriding  focus  is  on 
reducing  both  the  deployment  timeline  and  the  deployment  footprint.  The  Mission- 
Resource  Value  Assessment  Tool  (M-R  VAT)  is  designed  to  be  a  front-end  component  to 
the  ALP  architecture  that  will  reduce  both  the  timeline  and  the  footprint  by  enabling 
selection  of  the  best  value  assets  at  the  right  time. 

The  primary  goal  of  this  research  was  to  provide  a  USAF-wide  decision  support 
tool  in  support  of  the  M-R  VAT  that  quickly  identifies  the  best  possible  force  mixes 
based  upon  their  extrinsic  values.  The  secondary  goal  was  to  determine  the  most 
appropriate  mathematical  relationship  in  combining  extrinsic  and  intrinsic  values  into 
overall  force  mix  values.  To  accomplish  these  goals,  six  research  questions  were 
answered: 

1 .  Can  the  factors  affecting  force  mix  selection  in  response  to  theater  crises  be 
clearly  categorized  as  extrinsic  and  intrinsic,  and  can  these  categories  be  clearly 
defined? 
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2.  What  extrinsic  factors  are  important  to  campaign  planners  when  selecting 

aircraft  force  mixes  in  response  to  theater  crises? 

3.  How  are  these  factors  quantified,  and  what  are  the  relationships  between  them? 

4.  How  should  these  factors  and  the  relationships  between  them  be  modeled? 

5.  Can  the  research  results  be  applied  USAF- wide? 

6.  How  can  the  now-quantified  extrinsic  factors  be  effectively  combined  with  the 

intrinsic  factors  to  develop  an  overall  asset  set  utility  or  value? 

A  four-phase  methodology  was  implemented  to  address  these  six  research 
questions,  and  is  presented  in  Table  4  of  Chapter  IV. 

The  first  phase  of  research  consisted  of  in-person  interviews  with  campaign 
planning  experts  throughout  the  USAF,  using  the  Critical  Decision  Method,  and  content 
analysis  of  relevant  USAF  and  Department  of  Defense  publications  on  Joint  Planning. 

The  purpose  of  this  phase  was  to  establish  the  conceptual  separation  of  extrinsic  and 
intrinsic  decision  factor  categories,  to  define  these  categories,  and  to  elicit  all  of  the 
extrinsic  decision  factors  necessary  for  consideration  during  crisis  action  campaign 
planning.  The  experts  provided  unanimous  agreement  on  the  separation  of  extrinsic  and 
intrinsic  decision  factors,  helped  to  establish  definitions  for  each,  and  provided  22 
extrinsic  factors  important  to  force  mix  selection.  The  content  analysis  likewise 
presented  extrinsic  decision  factors,  as  well  as  a  set  of  intrinsic  factor  parent  issues  (see 
Figure  1,  page  19).  Phase  I  of  the  research  satisfied  questions  1  and  2. 

Phase  II  of  the  research  methodology  was  comprised  solely  of  a  literature  review 
of  the  Decision  Analysis  (DA)  field  to  determine  the  most  appropriate  DA  approach  with 
which  to  model  this  decision  situation.  Several  DA  methodologies  were  investigated  and 
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compared  to  the  requirements  of  this  decision  situation,  resulting  in  the  selection  of  Value 
Focused  Thinking  as  the  most  appropriate  technique.  This  effort  improves  upon  previous 
research  involving  this  decision  situation,  in  which  no  meaningful  comparisons  of  DA 
techniques  against  each  other  or  against  the  decision  situation  were  accomplished.  Phase 
II  results  satisfied  research  question  4. 

The  third  phase  of  methodology  incorporated  a  modified  Delphi  in-person  and 
email  study,  comprising  three  rounds.  The  three  rounds  of  the  modified  Delphi  study 
helped  to  establish  the  value  hierarchy  (necessary  to  creating  the  VFT  decision  support 
tool).  Additionally,  they  established  definitions  and  ranges  for  each  of  the  extrinsic 
decision  factors,  led  to  the  creation  of  64  single-dimension  value  functions  representing 
the  expert  judgements  of  the  study  participants,  and  generated  all  of  the  global  and  parent 
issue  weights  for  use  in  the  final  VFT  model.  Completion  of  Phase  III  satisfied  research 
question  3. 

Phase  IV  of  the  research  methodology  was  reserved  for  the  building  of  the  VFT 
decision  support  tool.  The  successful  averaging  of  individual  agency  single-dimension 
value  functions  (SDVF’s)  into  robust  USAF-wide  SDVF’s  was  the  final  step  in  satisfying 
research  question  5.  Further  satisfaction  of  research  question  5  is  the  inclusion  of  a 
family  of  models  within  the  single  VFT  framework,  allowing  flexibility  in  its  application. 
Finally,  the  development  of  a  multiplicative,  tie-breaking  relationship,  between  this 
extrinsic  VFT  model  and  an  intrinsically-scored  set  of  pareto  optimal  force  mixes, 
answers  research  question  6. 
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Summary  of  Research  Advancements 

The  primary  advancement  provided  by  this  research  over  previous  research  is  the 
USAF-wide  applicability  of  the  decision  support  tool.  The  six  research  questions 
notwithstanding,  USAF-wide  applicability  was  the  primary  goal  of  this  research.  It  is 
attained  via  the  diversity  and  breadth  of  experience  of  the  subject  matter  expert  pool, 
combined  with  the  averaging  of  individual  single-dimension  value  functions,  and  the 
inclusion  of  a  family  of  models  representing  five  agencies  and  the  USAF  overall. 

The  second  advancement  is  the  investigation  and  evaluation  of  a  large  number  of 
alternate  Decision  Analysis  (DA)  techniques.  Previous  research  had  attempted  no  such 
endeavor.  The  results  of  the  DA  investigation  validated  VFT  as  the  most  appropriate 
technique  for  modeling  and  addressing  this  decision  situation. 

The  establishment  of  an  accurate  mathematical  representation  of  the  extrinsic- 
intrinsic  value  relationship  constitutes  the  third  advancement.  With  intrinsic  value 
scored  and  a  set  of  pareto  optimal  force  mixes  established,  simple  multiplication  by  the 
respective  extrinsic  values  instantaneously  breaks  the  ties ,  leading  to  significantly 
reduced  force  mix  selection  timelines  and  the  simplified  selection  of  best  overall  force 
mixes. 

The  fourth  significant  advancement  is  the  straw  model  that  is  driven  by  both 
capability  requirements  and  operating  location  characteristics.  The  capabilities-driven 
aspect  allows  for  the  3x5  matrix  of  aircraft  types  and  aerospace  missions  to  generate 
candidate  force  mixes  that  are  relatively  equal  in  number  and  absolutely  equal  in  basic 
capability,  differing  only  by  their  suitability  scores.  The  operating  location  aspect  allows 
for  operating  location-specific  data  to  be  input  which  affects  the  value  of  candidate  force 
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mixes  while  also  allowing  for  different  operating  locations  to  be  scored  against  constant 
sets  of  force  mixes.  As  simply  a  test  set,  this  would  not  normally  warrant  consideration 
as  an  advancement.  However,  its  structure  readies  the  VFT  tool  now  for  decision  maker 
input  of  real-world  data  and  subsequent  real-world  force  mix  selection. 

Another  advancement  is  the  inclusion  of  96  single-dimension  value  functions 
(versus  the  16  of  previous  research)  representing  the  US  AF- wide  perspective  and 
perspectives  from  AEFC,  CADRE,  SouthAF,  CentAF,  and  ACC.  Combined  with  the 
inclusion  of  individual  issue  weights  as  determined  by  each  agency,  any  of  the  five 
agencies’  expertise  (or  the  USAF-wide  expertise)  can  drive  the  model. 

The  go/no  go  filter  is  a  conceptual  and  practical  improvement  that  not  only 
provides  feasible  sets  of  force  mixes  prior  to  extrinsic  scoring,  but  is  linked  by  formula  to 
operating  location  characteristics  and  straw  model  aircraft  type  characteristics. 

Finally,  this  research  remedied  a  flaw  in  the  original  VFT  decision  support  tool 
formulation  concerning  bomber  and  fighter  distances  from  targets  and  munitions  storage 
(MSA)  area  decision  factors.  Because  bombers  and  fighters  are  scored  separately  for 
distance  from  targets  and  MSA,  absence  of  either  fighters  or  bombers  in  a  given  force 
mix  could  lead  to  erroneous  force  mix  scoring.  For  example,  if  no  bombers  were 
included  in  a  candidate  force  mix,  its  straw  model  score  for  that  factor  would  be  zero. 

The  SDVF  formula  would  then  translate  that  zero  to  full  value  for  the  decision  maker 
because  bombers  would  be  zero  miles  from  targets.  The  force  mix’s  value  would  be 
artificially  inflated.  This  situation  is  remedied  by  allowing  the  global  weights  for  each  of 
these  factors  to  be  complimentary  between  fighters  and  bombers  based  upon  their 
respective  percentages  (in  number  of  aircraft)  of  the  total  force  mix. 
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Conclusions 


This  research  shows  that  the  collective,  subjective  judgements  of  subject  matter 
experts  can  be  combined  into  a  viable,  quantitative  decision  support  tool  for  the  selection 
of  best  force  mixes  during  crisis  action  campaign  planning.  Moreover,  it  shows  that 
dissimilar  single-dimension  value  functions  from  several  sources  can  be  effectively  and 
robustly  combined  into  overall  single-dimension  value  functions  that  accurately  represent 
the  views  of  all  involved.  The  technique  for  combining  single-dimension  value  functions 
was  not  found  in  literature  during  the  course  of  this  research,  and  may  be  unique  in  this 
application. 

An  important  aspect  of  the  developed  VFT  decision  support  tool  is  that  it  scores 
the  extrinsic  values  of  competing  force  mixes  almost  instantaneously,  in  a  format  that  is 
already  set  up  to  receive  the  intrinsically  scored  and  ranked  force  mixes  that  result  from 
concurrent  AFIT  research. 

Furthermore,  this  research  shows  that  the  VFT  decision  support  tool  has  a 
significant  impact  on  the  rankings  of  force  mixes  that  have  been  previously  scored  and 
ranked  intrinsically. 

The  bottom-line  value  of  this  VFT  decision  support  tool  is  that,  given  a  set  of 
intrinsically  scored  force  mixes,  it  can  almost  instantly  produce  overall  values  for  each 
candidate  mix  and  subsequently  break  the  ties  among  force  mixes  that  have  been 
optimized. 
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Limitations 


The  primary  limitation  of  this  decision  support  tool  is  that  it  scores  only  the 
extrinsic  value  of  candidate  force  mixes.  Although  it  is  formatted  to  produce  overall 
force  mix  values,  it  must  do  so  with  force  mixes  that  have  already  been  scored  and 
ranked  intrinsically. 

Another  limitation  is  the  size  of  the  subject  matter  expert  pool  and  the  number  of 
agencies  involved  in  the  Phase  III  portion  of  study.  This  research  focused  on  the 
opinions  of  experts  within  the  continental  United  States,  ignoring  expertise  that  might  be 
provided  from  planners  in  overseas  commands.  The  possibility  exists  that  inputs  from 
overseas  commands  could  change  the  single-dimension  value  functions  and  global 
weights  within  the  VFT  model.  Although  sensitivity  analyses  might  prove  such  inputs 
inconsequential,  the  limitation  exists  nonetheless. 

Recommendations 

It  is  recommended  that  this  VFT  decision  support  tool  be  fully  incorporated  into 
the  M-R  VAT  as  a  front  end  force  mix  value  determinant  for  the  Advanced  Logistics 
Project.  It  provides  an  avenue  for  campaign  planners  to  quickly  evaluate  the  effects  of 
extrinsic  considerations  on  candidate  force  mixes.  Moreover,  the  model’s  format  and 
formulas  are  consistent  with  ongoing  AFIT  research  and  qualify  it  as  the  template  upon 
which  the  M-R  VAT  can  be  based. 

The  VFT  decision  support  tool  can  run  now  in  a  real-world  environment. 
However,  it  is  built  upon  notional  data,  necessitating  the  introduction  of  actual  extrinsic 
data  before  it  can  be  applied  operationally. 
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Further  Research 


The  additional  research  most  necessary  to  the  success  of  this  VFT  decision 
support  tool  is  the  collection  of  real-world  data  as  it  relates  to  the  extrinsic  value  of 
individual  weapons  platforms.  For  example,  what  is  the  ratio  of  consumption  to  resupply 
for  an  F-16C  assigned  to  contingency  operations  at  Base  X,  or  what  is  the  Munitions 
Storage  Area  requirement  for  an  A- 10? 

Further  research  is  also  necessary  to  determine  the  most  effective  integration  of 
this  decision  support  tool  into  the  M-R  VAT,  including  its  integrative  programming  and 
the  development  of  a  graphical  user  interface. 

Finally,  further  investigation  is  necessary  to  determine  the  relationships  among 
individual  factors  within  and  between  the  intrinsic  and  extrinsic  categories.  Some  may 
not  fall  entirely  within  one  category  or  the  other.  Distinction  between  the  categories  (and 
appropriate  definitions)  has  been  established.  However,  there  is  a  possibility  that  some  of 
the  individual  factors  may  fall  within  both  categories. 

Summary 

The  Value  Focused  Thinking  Decision  Support  Tool  developed  by  this  research 
provides  a  method  to  quickly  evaluate  candidate  force  mixes  for  contingency 
deployments  based  upon  factors  that  are  specific  to  the  situation  (extrinsic)  rather  than  to 
the  basic  capabilities  of  the  weapons  platforms  themselves  (intrinsic).  It  provides  a 
means  for  almost  instantaneously  computing  overall  force  mix  values  once  those  force 
mixes  have  been  valued  intrinsically. 


The  support  tool  was  developed  with  the  expertise  of  campaign  planning  experts 
from  Headquarters  USAF,  Air  University,  Air  Combat  Command,  Southern  Air  Forces 
Command,  Central  Air  Forces  Command,  the  Air  Expeditionary  Forces  Center,  Joint 
Forces  Command,  Air  Force  Special  Operations  Command,  RED  FLAG,  and  Fighter 
Weapons  School. 

The  model  incorporates  combined  single-dimension  value  functions,  as  well  as 
value  functions  and  weights  representing //ve  agencies.  This  combination  enhances 
USAF-wide  applicability.  It  was  demonstrated  and  validated  using  a  capabilities  and 
operating  location  driven  straw  model  of  5 1  candidate  force  mixes.  The  validation 
showed  all  weights,  value  functions,  and  the  overall  model  to  be  robust  during  internal 
and  external  sensitivity  analyses. 

This  research  and  its  resultant  VFT  decision  support  tool  present  a  number  of 
advancements  over  previous  research  in  this  area,  providing  a  sound  building  block  upon 
which  ongoing  AFIT  research  in  support  of  the  ALP  program  can  be  based. 

This  work  is  a  first  step  in  quantifying  the  art  of  force  selection.  Such 
quantification  can  allow  planners,  regardless  of  experience  level,  to  act  and  decide  crisis 
action  planning  issues  in  a  manner  consistent  with  the  best  planners  in  today’s  USAF. 
This  in  turn  can  generate  consistency  of  planning  competence  across  all  USAF  agencies. 
Furthermore,  these  consistent,  expert-level  decisions  could  be  made  in  fractions  of  the 
time,  reducing  deployment  timelines  and  footprints  while  enhancing  the  USAF’s  most 
core  competency — warfighting. 
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Appendix  A:  Bullet  Background  Paper 


Purpose  of  this  background  paper:  to  provide  the  respondent  with  an  overview  of  the 
topics  of  discussion  to  be  addressed  during  the  interview. 

Purpose  of  the  interview :  to  obtain  knowledge  and  understanding  of  the  process  and 
factors  considered  when  selecting  certain  combat  aircraft  asset  sets  for  theater 
deployment. 

-  USAF  now  operates  under  the  expeditionary  concept 

—  Deploy  small  contingent  of  aircraft,  36  fighter  and  2-6  bomber  assets  per  set 

—  May  be  expected  to  commence  aerospace  missions  immediately  upon  arrival 

—  Campaign  will  require  several  types  of  aerospace  missions,  defined  in  AFM  1-1 

-  Our  purpose  is  to  look  ahead  beyond  current  prepackaged forces 

—  A  tailored  force  for  each  scenario 

—  Bring  everything  you  need  but  only  what  you  need 

—  Calculate  time-phase  requirements,  spin  up  to  fight  from  day  zero. .  .no  more 
“closure”  of  forces 

-  Please  walk  through  a  complete  planning  process  for  expeditionary  force  contingency 

-  Realizing  there  are  several  layers  of  planning  and  decision-making,  your  views  at 

your  level  are  of  prime  importance  in  this  study 

—  What  is  the  fundamental  process  of  selecting  aircraft  types  to  support  a  campaign? 

—  In  determining  the  proper  aircraft  mix  to  be  deployed  and  in  planning  for  their 
deployment,  what  factors  do  you  look  at? 

-  Focus  on  identified  factors 

—  What  makes  these  important? 

—  Is  there  anything  that  may  affect  this  factor. .  .underlying  sub-factors? 

—  How  do  these  factors  relate  to  each  other  and  the  overall  process? 

-  Conclusion 

—  Any  other  factors  that  have  not  been  identified  that  you  feel  are  important,  no  matter 
level 

—  Any  important  topics  we  did  not  discuss? 

—  Any  contacts  you  know  of  that  I  should  talk  to  regarding  this  research? 

—  Soon,  a  second  phase  of  research  will  begin.  Your  continued  support  will  be  greatly 
appreciated 

—  Second  phase  will  be  quickly-accomplished  email  questionnaires  to  determine 
values  and  relative  weights  of  the  factors  we  have  identified  today — your  support 
will  be  crucial 
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-  Points  of  contact  for  this  research 

—  1st  Lt  Paul  Filcek,  AFIT  (Paul.Filcek@afit.af.mil') 

--  Lt  Col  Alan  Johnson,  AFIT  (Alan.Johnson@afit.af.mil),  DSN  255-3636  ext  4284 

—  Maj  Stephen  Swartz,  AFIT  (Stephen.Swartz@afit.af.mil) 

—  Advanced  Logistics  Project  (ALP)  Website  (www.darpa.mil/iso/alp) 
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Appendix  B:  Interview  Script 


INTRODUCTION 

Good  afternoon.  Sir.  Thank  you  for  taking  the  time  to  meet  with  me  today.  I  am 
conducting  research  into  the  critical  factors,  or  issues,  that  influence  the  selection  and  use 
of  combat  aircraft  types  for  contingency  deployments.  The  purpose  of  this  interview  is  to 
elicit  your  knowledge  and  understanding  of  the  overall  process  as  an  expert  in  the  field. 

The  information  you  provide  will  be  combined  with  that  of  other  experts  and 
reported  in  aggregate.  Your  information  will  be  confidential  in  that  it  will  not  be  traced 
specifically  to  you.  Only  my  observers  and  myself  will  have  access  to  any  identifiable 
information. 

Understanding  this  obligation  of  confidentiality,  I  would  like  to  ask  your 
permission  to  record  this  interview.  The  sole  purpose  of  the  recording  is  to  overcome  my 
own  lack  of  shorthand  skills.  Furthermore,  the  tape  will  be  reused  for  subsequent 
interviews. 

I  am  concerned  with  not  taking  more  than  45  minutes  of  your  time,  so  please 
excuse  me  in  advance  for  watching  the  clock  and  for  pushing  the  pace  if  we  slow  down. 
However,  I  must  stress  that  the  interview  is  largely  unstructured,  and  any  discussions 
within  the  context  of  the  interview  are  encouraged.  Do  you  have  any  questions  before  we 
star? 

BACKGROUND 

To  begin  our  discussion  of  the  process  and  the  factors,  I  would  like  to  first  set  the 
deployment  stage.  As  you  know,  we’ve  moved  to  the  expeditionary  concept  in  which  the 
plan  is  to  deploy  about  36  fighter  and  2-6  bomber  assets  and  their  necessary  support 
equipment.  These  newly-arrived  forces  will  be  expected  to  commence  engagement 
immediately  upon  arrival,  and  such  engagement  of  the  enemy  may  require  several  types 
of  aerospace  missions  as  defined  in  AFM  1-1  and  determined  by  the  theater  commander. 

Research  is  currently  being  conducted  in  an  effort  to  assist  theater  commanders 
and  campaign  planners  in  selecting  sets  of  aircraft  assets  that  will  provide  the  best  utility 
to  the  commander,  based  upon  the  theater  commander’s  time-phased  aerospace  mission 
requirements.  The  research  scenario  is  that,  using  a  developed  mathematical  program, 
asset  sets  will  be  identified  to  provide  the  commander  with  the  best  match-up  of  aircraft 
assets  to  the  required  missions.  Today,  these  match-ups  are  based  upon  a  “prepackaged 
set”  of  forces,  with  the  capabilities  of  each  individual  aircraft  matched  to  specific 
aerospace  missions.  Of  course,  we  know  that  we  look  at  more  than  just  these  absolute 
mission  capabilities  when  planning  a  deployment.  There  are  many  factors  of  a 
deployment  scenario  that  may  make  the  selection  of  one  aircraft  type  preferred  over 
another  type,  or  may  negate  the  possibility  of  selecting  one  or  more  types.  Understanding 
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of  the  overall  process  is  the  focus  of  today’s  interview.  I  am  going  to  ask  you  a  series  of 
questions  aimed  at  obtaining  your  views  and  insights. 

I  must  finish  the  introduction  by  asking  you  not  to  hold  back ;  have  confidence  in 
your  thoughts  and  feelings  on  these  issues.  I  am  going  to  many  offices  in  search  of  a 
model  that  applies  AF-wide  and  your  experience  is  very  important  in  the  intended  scope 
of  this  research! 

Name: 

Duty  Position: 

Office  Symbol: 

Email  address: 

DSN: 

1 .  How  many  years  of  experience  do  you  have  in  the  Air  Force? 

2.  With  which  MDSs  do  you  have  operational  experience?  How  many  years  for  each? 

3.  For  which  theaters  or  commands  have  you  done  campaign  or  contingency  planning? 

4.  How  many  years  experience  for  each? 


Those  are  the  required  demographic  questions.  Now  we  can  move  on  to  deployment 
factors... 

5.  I  would  like  you  to  walk  me  through  the  planning  process  for  a  contingency 
deployment,  one  that  would  be  suited  to  an  expeditionary-sized  deployment.  I  know 
there  are  many  levels  of  decision-making,  but  I  am  interested  in  your  views  at  your  level 
of  the  process.  In  determining  the  proper  aircraft  asset  sets  to  send,  what  factors  do  you 
look  at?  Please  consider  the  situations  and  theaters  with  which  you  are  most  familiar  and 
have  the  greatest  recall. 


6.  Are  there  any  more  details  or  factors,  no  matter  how  insignificant  you  may  feel  they 
are? 


7.  Are  these  all  of  the  factors  you  can  think  of  at  this  time?  Please  tell  me  of  any  others. 
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8.  I  have  extracted  numerous  factors  from  content  analysis  of  regulations/guidance.  Do 
you  agree  with  the  existence  of  these?  ( show  to  subject,  compare  and  contrast,  integrate 
if  possible).  Now  show  Buzo ’s.  Please  evaluate  these  factors  in  like  manner. 

9.  We’ve  established  quite  a  list.  It  is  crucial  to  have  each  factor  independent  of  the 
other  factors.  Are  each  of  the  factors  listed  independent  of  the  others,  or  does  one  or 
some  depend  on  one  or  more  of  the  others  in  its  evaluation?  I  realize  that  each  affects  the 
total  outcome,  but  does  a  change  in  any  factor  affect  another  factor  directly?  (if  A 
changes,  does  B  change  because  A  did?) 


10.  Can  these  factors  be  divided  into  two  groups,  one  group  pertaining  to  the  quality  of 
the  force  mix  and  another  group  pertaining  to  the  situation?  Put  another  way,  are  some 
factors  extrinsic  to  the  mission  capability  of  an  asset  set  and  other  factors  intrinsic  to  that 
capability? 


1 1 .  How  would  you  define  an  extrinsic  factor?  An  intrinsic  factor?  Here  are  my 
definitions  (after  the  response).  What  do  you  think? 

12.  Finally,  let’s  define  each  of  these  extrinsic  factors.  What  changes  should  I  make  to 
some  of  the  definitions  I’ve  already  got? 


Thank  you  VERY  much  for  the  interview!  Soon,  I  will  begin  the  second  phase  of  this 
research  that  will  consist  of  email  questionnaires.  Your  participation  during  this  phase 
will  be  vitally  important  in  determining  a  decision  support  tool  that  is  the  main  point  of 
the  study.  The  next  questionnaires  will  be  designed  for  user-friendliness  and  fast 
accomplishment  in  order  to  keep  the  demand  on  your  time  as  low  as  possible  and  to  keep 
your  interest  in  the  study. 

Again,  it  has  been  a  sincere  pleasure.  I  thank  you  very  much  and  look  forward  to  your 
continued  support  over  the  next  few  weeks. 
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Appendix  C  :  Extrinsic  Decision  Factors 


Constraints: 


1 .  Politics — Desired  Force  Mix  Allowed? 

2.  Operating  Location — Runway  Length  and  Width  Adequate? 

3.  Operating  Location — Ramp  Space  Sufficient? 

4.  Operating  Location — Runway  and  Ramp  Weight  Capacity  Sufficient? 

5.  Operating  Location — No  IFR,  Distance  Less  Than  700  (Fighters)  and  3000 
(Bombers)  Miles  from  Targets? 

6.  Desired  Force  Mix  Available? 


Value  Function  Factors  (to  be  Weighted  and  Scored): 


1 .  Multi-National  Compatibility 

2.  Beddown  Location — Fighter  Distance  (From  Targets,  Refueled) 

3.  Beddown  Location — Bomber  Distance  (From  Targets,  Refueled) 

4.  Munitions  Storage  Area — Fighter 

5.  Munitions  Storage  Area — Bomber 

6.  Ability  to  Resupply 

7.  Petroleum,  Oil,  and  Lubricants  (POL)  Storage/Delivery 

8.  Tanker  Support  Provided 

9.  Operating  Location — Topography 

10.  Operating  Location — Climate 

1 1 .  Mission  Capable  Status  of  Force  Mix 

12.  Inter-mission  Flexibility  of  Force  Mix 

13.  Force  Mix  Utilization  Rate 

14.  USAF  vs.  Enemy  Capability  Ratio 

15.  Enemy  Intelligence,  Surveillance,  and  Reconnaissance  Impact 

16.  Enemy  Interference  Impact  (at  operating  location) 
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Appendix  D:  Extrinsic  Decision  Factors — Definitions  and  Ranges 
(those  weighted  and  scored  as  value  functions) 


Title:  Multi-National  Compatibility 

Description:  The  extent  to  which  host  nation  assets,  to  include  combat  aircraft, 
equipment,  spares,  and  ground  support  equipment  can  be  incorporated  along  side  and  into 
USAF  combat  aircraft  [Buzo,  2000:  105]. 


Range  of  Measurement:  0-100  percent  for  a  monotonically  increasing  function.  At  100 
percent  compatibility,  the  host  nation’s  candidate  operating  location  for  US  forces 
maintains  and  operates  fully  compatible  assets  for  100  percent  of  the  assets  in  the 
candidate  force  mix.  USAF  combat  aircraft  can  be  incorporated  into  the  operating 
location  without  bringing  in  test  equipment,  spares  or  excessive  maintenance  equipment. 
At  0  percent,  none  of  the  host  nation  assets  are  compatible  with  those  of  the  candidate 
force  mix.  In  this  case,  campaign  planners  would  need  to  arrange  for  sending  a  complete 
supply  of  testing  equipment,  spares,  and  maintenance  and  repair  equipment  for  all  combat 
aircraft  sent  to  the  operating  location.  The  entire  range  of  percentages  of  host  nation 
compatibility  is  possible. 


Title:  Beddown  Location — Fighter  Distance  from  Targets,  Refueled 

Description:  The  effectiveness  of  an  individual  aircraft  type  within  a  candidate  force  mix 
is  dependent  upon  its  distance  from  the  operating  location  to  its  primary  mission  targets. 

Range  of  Measurement:  0-3000  miles.  Given  availability  of  tanker  support  to  provide 
In-flight  refueling  (IFR),  this  is  a  measure  of  time-to-target  as  much  as  distance — 
becoming  an  aircrew  effectiveness  limitation  (and  loiter  time  limitation)  as  the  distance 
from  the  operating  location  to  the  primary  target  areas  approaches  3000  miles.  The  entire 
range  of  0  to  3000  miles  is  possible. 
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Title:  Beddown  Location — Bomber  Distance  from  Targets,  Refueled 


Description:  The  effectiveness  of  an  individual  bomber  aircraft  type  within  a  candidate 
force  mix  is  dependent  upon  its  distance  from  the  operating  location  to  its  primary 
mission  targets. 

Range  of  Measurement:  0-7000  miles.  This  measure  is  identical  to  that  of  Fighter 
aircraft,  with  the  only  difference  being  the  greater  range  of  bomber  aircraft. 


Title:  Ability  to  Resupply 

Description:  Continuation  of  aerospace  missions  from  the  operating  location  is 
dependent  upon  the  ability  of  USAF  forces  to  resupply  equipment,  materials,  and  spares 
as  necessary.  The  candidate  operating  location  can  impact  resupply  capability  and 
combine  with  individual  aircraft  resupply  requirements  to  affect  overall  mission 
accomplishment.  Aircraft  that  require  smaller  amounts  of  POL,  munitions,  and  other 
equipment  and  spares  will  have  a  higher  independent  value  than  aircraft  that  require  more 
resources,  especially  if  resources  are  difficult  to  come  by  at  the  candidate  operating 
location. 

Range  of  Measurement:  0- 1,  ratio  of  resupply  to  consumption.  As  the  ratio  of  resupply 
to  consumption  increases  to  1,  USAF  forces’  ability  to  resupply  the  operating  location 
equals  the  consumption  at  that  location.  As  the  ratio  approaches  0,  consumption  rates  far 
exceed  supply  at  the  operating  location.  Ratios  above  1  are  not  expected  because  they 
would  imply  an  excessive  application  of  resources  in  time  of  crisis.  The  entire  range  of  0 
-1  is  possible. 


Title:  Munitions  Storage  Area — Fighter 

Description:  Continued  combat  fighter  operations  depend  on  the  operating  location’s 
ability  to  store  and  deliver  adequate  munitions  in  a  timely  manner.  Fighters  can  bring  to 
theater  only  one  load  of  munitions  each.  Additional  aerospace  missions  require  the 
operating  location  to  store  and  deliver  munitions. 

Range  of  Measurement:  0-100  percent  sorties  supported.  At  100  percent  sorties 
supported,  the  operating  location  can  store  and  deliver  all  of  the  munitions  necessary  to 
conduct  the  required  aerospace  operations.  At  0  percent  sorties  supported,  the  operating 
location  can  store  virtually  no  munitions,  and  the  only  munitions  available  are  those 
loaded  upon  each  aircraft  upon  departure  from  its  home  station.  The  entire  range  of 
munitions  storage  and  delivery  is  possible. 
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Title:  Munitions  Storage  Area — Bomber 


Description:  Same  as  Munitions  Storage  Area — Fighter 

Range  of  Measurement:  0-100  percent  sorties  supported.  Same  as  Munitions  Storage 
Area — Fighter 


Title:  Petroleum,  Oil,  and  Lubricants  (POL)  Storage  and  Delivery 

<r 

Description:  The  operating  location’s  POL  system  includes  storage  and  supply  from  both 
military  and  host  nation  contracted  sources.  POL  capability  for  a  particular  force  mix 
must  take  into  consideration  the  per-sortie  POL  requirements  of  each  individual  aircraft 
in  the  force  mix  mix  as  well  as  the  daily  sortie  requirements  for  each  type  of  aircraft 
[Buzo,  2000:  107], 

Range  of  Measurement:  0-100  percent  sorties  supported.  At  100  percent  sorties 
supported,  the  operating  location’s  POL  system  can  store  and  deliver  sufficient  POL  to 
sustain  100  percent  of  the  projected  operational  sorties.  At  0  percent  POL  capability,  the 
operating  location  could  support  no  sorties.  Of  course,  this  would  disqualify  the 
operating  location  as  a  candidate  base.  The  entire  range  of  POL  capability  is  possible. 


Title:  Tanker  Support  Provided 

Description:  Depending  upon  the  operating  location’s  distance  from  enemy  targets  and 
the  requirements  of  individual  aircraft  and  aircraft  types  within  a  force  mix,  tanker 
support  may  be  required  or  highly  desirable. 

Range  of  Measurement:  0-100  percent  sorties  supported.  At  100  percent  sorties 
supported,  enough  tankers  are  available  and  included  in  the  deployment  plan  to  support 
all  of  the  aerospace  missions  necessary.  At  0  percent  sorties  supported,  no  tanker  support 
is  available  for  operations.  The  entire  range  is  possible. 
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Title:  Operating  Location — Topography 


Description:  The  extent  to  which  natural  and  man-made  land  formations  in  and  around 
the  operating  location  affect  the  ability  of  individual  aircraft  within  a  force  mix  to  take¬ 
off,  land,  and  conduct  operations. 

Range  of  Measurement:  0-100  percent  mission  degradation.  At  100  percent  mission 
degradation,  no  sorties  can  fly  in  or  out  of  the  operating  location  due  to  topographical 
restrictions.  Such  degradation  is  a  theoretical  maximum,  since  its  occurrence  would 
necessitate  relocation  of  combat  force  mixes.  At  0  percent  degradation,  no  sorties  are  lost 
to  topographical  restrictions.  The  entire  range  of  scores  is  possible. 


Title:  Operating  Location-Climate 

Description:  The  extent  to  which  heat,  humidity,  wind,  and  adverse  weather  conditions 
affect  the  ability  of  force  mixes  to  conduct  operations  at  or  around  the  operating  location. 

Range  of  Measurement:  0-100  percent  mission  degradation.  As  in  Topography  above, 
100  percent  degradation  would  mean  that  no  sorties  could  be  generated,  but  due  to 
weather  conditions  here.  At  0  percent  degradation,  all  sorties  could  be  conducted  without 
weather  interruption  or  disruption.  The  entire  range  is  possible. 


Title:  Mission  Capable  Rate  of  Force  Mix 

Description:  Extent  to  which  aircraft  in  a  candidate  force  mix  are  operationally  capable 
of  performing  aerospace  missions  as  designated  by  the  theater  or  component  commander. 

Range  of  Measurement:  0-100  percent  average  mission  capable  rate.  At  100  percent 
mission  capability,  all  of  the  aircraft  in  the  candidate  force  mix  have  100  percent  mission 
capable  rates.  At  0  percent  mission  capable,  none  of  the  aircraft  within  the  force  mix  are 
mission  capable.  Both  extremes  are  near-theoretical,  short  of  pre-deployment  generation 
or  the  grounding  of  a  fleet  for  one  time  inspection  or  time  compliance  technical  order.  At 
50  percent  mission  capable,  some  aircraft  within  the  force  mix  could  be  higher  while 
some  are  lower.  All  values  within  the  range  are  possible. 
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Title:  Inter-mission  Flexibility 


Description:  Extent  to  which  individual  aircraft  types  within  a  candidate  force  mix  can 
perform  all  of  the  aerospace  missions  defined  in  AFM  1-1  and  AFDD  2-1.  The  relative 
worth  of  the  force  mix  would  be  highest  if  it  could  perform  all  of  the  missions  required 
by  the  theater  or  component  commander  [Buzo,  2000:  110]. 

Range  of  Measurement:  capability  to  perform  0-100  percent  of  required  missions.  This 
is  inter-mission  flexibility.  Therefore,  at  0  percent,  the  force  mix  can  perform  one,  but 
only  one,  type  of  aerospace  mission.  At  1 00  percent,  the  force  mix  can  perform  all  of  the 
potentially-desired  aerospace  missions  as  dictated  by  the  theater  or  component 
commander.  It  is  important  to  note  that  the  force  mix — as  a  collection  of  aircraft 
designs — can  perform  all  of  the  missions.  No  single  aircraft  within  the  mix  need  be 
capable  of  such  flexibility.  All  of  the  values  within  the  range  are  possible. 


Title:  Utilization  Rate  of  Force  Mix 

Description:  Extent  to  which  aircraft  within  the  force  mix  are  to  be  used  in  the 
performance  of  primary  missions.  This  does  not  preclude  show  of  force  deployments. 

Range  of  Measurement:  0—100  percent  utilization  rate.  At  0  percent  utilization  rate, 
none  of  the  aircraft  within  the  force  mix  are  to  be  used  during  the  contingency.  This 
would  be  an  example  of  a  show  of  force  deployment.  As  utilization  rates  approach  100 
percent,  degradation  of  mission  capability  is  highly  probable.  Therefore,  this  measure  is 
monotonically  decreasing.  The  higher  the  utilization  rate,  the  lower  the  value  to  the 
decision  maker.  All  of  the  values  within  the  0-100  percent  range  are  possible. 
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Title:  USAF  versus  Enemy  Capability 


Description:  The  comparison  of  enemy  assets  to  USAF  combat  aircraft  assets  as 
contained  in  the  candidate  force  mix.  Assuming  US  forces’  intelligence  can  discern  the 
extent  of  the  enemy’s  capability,  this  measure  shows  the  relative  worth  of  the  amount  of 
aircraft  assets  sent  to  the  operating  location.  One  issue  addressed  within  the  function  is 
the  number  of  USAF  combat  aircraft  sent.  The  second  issue  is  the  capabilities  of  the 
enemy  to  counter  the  USAF  force  mix  [Buzo,  2000:  111]. 

Range  of  Measurement:  0:1  to  3:1,  ratio  of  USAF  to  enemy  forces.  As  the  ratio 
approaches  0:1,  USAF  forces  are  severely  outmanned.  At  1:1,  USAF  forces  are  equally 
offset  by  enemy  forces.  And  as  the  ratio  approaches  3:1,  USAF  forces  gain  a  decided 
advantage.  The  entire  range  of  ratios  is  possible,  but  as  the  USAF  advantage  passes  3:1, 
no  additional  value  is  attained  and  scarcity  of  resources  reserved  for  other  engagements 
becomes  an  issue. 


Title:  Enemy  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  Impact 

Description:  Addresses  the  enemy’s  ability  to_see  or  obtain  information  on  US  activity  at 
the  operating  location.  Enemy  ISR  could  be  obtained  through  aerial  or  satellite 
photography,  local  sympathizers  monitoring  US  activity,  or  spies  within  the  host  nation 
working  within  or  around  the  operating  location  [Buzo,  2000:  112].  Also  necessarily 
addressed  by  this  factor  is  the  sensitivity  to  ISR  of  certain  aircraft  types  within  candidate 
force  mixes,  and  the  tradeoffs  of  such  sensitivity  with  the  enemy’s  ISR  capabilities  at  the 
candidate  operating  location. 

Range  of  Measurement:  0- 100  percent  degradation  of  mission  due  to  enemy  ISR.  This 
is  not  a  measure  of  exposure,  but  rather  a  measure  of  the  negative  impact  of  such 
exposure.  At  0  percent  mission  degradation,  all  USAF  combat  sorties  are  conducted  as 
scheduled  and  no  sensitive  assets  are  compromised.  At  100  percent  degradation,  no 
sorties  can  go  as  scheduled  due  to  enemy  ISR  and/or  major,  crippling  information  or 
intelligence  has  been  surrendered  to  the  enemy.  All  of  the  values  within  the  range  are 
possible. 
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Title:  Enemy  Interference 


Description:  Addresses  the  enemy’s  ability  to  interfere  with  US  operations  at  the 
operating  location.  Interference  is  any  action  on  the  part  of  the  enemy  that  could  hamper 
US  combat  operations.  The  interference  could  range  from  relatively  harmless 
demonstrations  and  propaganda  to  bomb  threats  and  actual  sabotage  upon  USAF  assets 
and  harm  to  personnel.  This  necessarily  includes  defensibility  of  the  operating  location 
from  all-out  attack  upon  the  base,  considering  the  strength  of  opposing  forces  in  the 
vicinity  of  the  operating  location  as  well  as  the  projected  defense  forces  in  place  and/or  to 
be  sent. 

Range  of  Measurement:  0-100  percent  degradation  of  mission.  At  0  percent 
degradation  of  mission,  no  enemy  interference  of  any  kind  is  to  be  encountered  at  the 
candidate  operating  location.  At  100  percent  degradation,  enemy  interference  will  halt  all 
combat  aerospace  operations  at  the  candidate  operating  location.  The  entire  range  of 
values  within  enemy  interference  is  possible. 
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Appendix  E:  Rank-Ordering  Sheets  (Delphi  Round  1) 


THE  ISSUE  AND  ITS  MEASUREMENT _ YOUR  RANK 

Multi-National  Compatibility: . 

Compatibility  between  in-place  host  nation  and  US  assets  (0-100%) 

Beddown  Location — Fighter  Distance  (refueled): . 

Deployed  location’s  distance  from  enemy  targets  (0-3000  miles) 

Beddown  Location — Bomber  Distance  (refueled): . 

Deployed  location’s  distance  from  enemy  targets  (0-7000  miles) 

Ability  to  Resupply: . 

Ratio  of  consumption  to  resupply  (0  to  1) 

Allow  Assets  In: . 

Yes  or  No  constraint 

International  Politcs: . 

Yes  or  No  constraint 

Intra-National  Politics: . 

Yes  or  No  constraint 

Inter-Service  Politics: . . 

Yes  or  No  constraint 

Intra-Service  Politics: . 

Yes  or  No  constraint 

Operating  Location — Runway  Length  and  Width: . 

Yes  or  No  constraint 

Operating  Location — Runway/Ramp  Weight  Capacity: . . 

Yes  or  No  constraint 

Operating  Location — Ramp  Space: . 

Yes  or  No  constraint 

Munitions  Storage  Area — Fighter: . 

Percent  of  fighter  sorties  supportable  per  day  (0-100%) 
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THE  ISSUE  AND  ITS  MEASUREMENT 


YOUR  RANK 


Munitions  Storage  Area — Bomber: . 

Percent  of  bomber  sorties  supportable  per  day  (0-100%) 

Petroleum,  Oil,  Lubricants  (POL)  Storage/Delivery: . 

Percent  of  force  mix  sorties  supportable  per  day  (0-100%) 

Tanker  Support  Available: . 

Percent  of  force  mix  sorties  supportable  per  day  (0-100%) 

Operating  Location — Topography: . 

Percent  degradation  of  Mission  (0-100%) 

Operating  Location — Climate: . 

Percent  degradation  of  Mission  (0-100%) 

Asset  Inter-Mission  Flexibility: . 

Percent  of  the  required  aerospace  missions  force  mix  is  capable  (0-100%) 

Force  Mix  Utilization: . 

Average  expected  utility  of  all  aircraft  in  force  mix  (0-100%) 

Availability  of  Force  Mix: . 

Average  mission  capable  rate  of  aircraft  in  force  mix  (0-100%) 

Enemy  vs.  USAF  capability: . 

Ratio  of  enemy-to-allied  forces  (0:1  to  3:1) 

Enemy  Intelligence,  Surveillance,  Reconnaissance  (ISR)  Impact: . 

Negative  sortie  impact  of  operations  exposure  to  enemy  (0-100%) 

Enemy  Interference: . 

Percent  degradation  of  mission  (0-100%) 

Is  Force  Mix  Available  to  Deploy? . . 

Yes  or  No  Constraint 
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Appendix  F:  Instructional  Paper  on  Issue  Weighting 


1 .  The  purpose  of  the  upcoming  research  questionnaire  is  to  obtain  opinions  and 
knowledge  on  the  relative  weightings  between  the  campaign  specific  issues  identified 
during  previous  interviews  and  research.  The  purpose  of  this  background  paper  is  to 
provide  an  understanding  of  the  process  of  swing  weighting  and  to  present  an  example  of 
such. 

-  Previous  rounds  of  this  study  have  determined  relative  ranking  of  Campaign  Specific 
issues 

—  Ranking,  based  on  level  of  importance,  of  issues  has  been  determined 

-  The  actual  questionnaire  will  consist  of  comparing  15  sets  of  two  separate  campaign 
specific  issues  to  determine  relative  weights  of  all  issues 


2.  To  understanding  how  comparisons  will  be  conducted,  an  example  is  provided.  When 
deciding  to  purchase  a  car,  many  features  must  be  considered.  Consider  two  cars;  Car 
‘A’  has  all  3  of  the  safety  features  required  (i.e.  anti-lock  brakes,  traction  control,  and 
dual  air  bags),  and  costs  $25,000.  Car  ‘B’  has  1  of  the  3  safety  features  (dual  air  bags) 
needed,  and  costs  $20,000.  For  simplicity,  other  features  will  be  disregarded. 

-  First,  the  value  of  these  issues  (safety  features  available  and  cost)  based  upon  personal 
requirements  must  be  identified. 

—  Determine  ‘value’  for  the  price  of  the  car,  based  on  personal  requirements 

—  Paying  $20,000  or  less  is  optimal,  resulting  in  a  value  of  1 .0  at  $20,000 

—  Paying  $30,000  or  more  is  bad,  resulting  in  a  value  of  0  at  $30,000 

—  Value  decreases  linearly  between  $20,000  and  $30,000  ($25,000  results  in  a  value 
of  0.5) 
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—  Determine  ‘value’  for  having  safety  features,  based  on  personal  requirements 

—  Having  all  3  safety  features  is  optimal,  resulting  in  a  value  of  1.0  for  all  three 

—  Having  none  of  the  safety  features  is  very  bad,  resulting  in  a  value  of  0. 

—  Value  decreases  equally  between  3  to  no  features  (1  feature  results  in  a  value  of 
0.33,  2  features  result  in  a  value  of  0.66) 

--  Using  the  example,  ‘Safety’  value  of  Car  ‘A’  is  1.0,  and  ‘Cost’  value  is  0.5 

--  ‘Safety’  value  of  Car  ‘B’  is  0.33,  ‘Cost’  value  is  1.0 

-  Next,  relative  weighting  between  two  issues  must  be  determined 

--  It  is  determined  that  safety  features  are  more  important  than  cost 

—  Compare  ‘range  of  measurement’  of  safety  (0  to  3  features)  to  ‘range  of 
measurement’  of  cost  ($20,000  to  $30,000) 

—  As  safety  features  more  important  than  cost,  compare  cost  to  safety  by  asking  the 
following  question: 

WHAT  PORTION  OF  THE  ‘RANGE  OF  MEASUREMENT’  OF  SAFETY 
EQUALS  THE  ‘RANGE  OF  MEASUREMENT’  OF  COST? 

—  Question  determines  level  of  concern  over  reduced  value  between  two  issues 

—  If  cost  swings  from  $20,000  to  $30,000  (best  value  to  worst  value),  what  would  be 
the  equivalent  loss  in  safety  features. 

—  Based  on  personal  belief,  it  is  determined  a  swing  in  cost  from  $20,000  to  $30,000 
(best  value  to  worst  value)  is  comparable  (same  level  of  unhappiness  is  felt)  to  a 
swing  in  losing  1  safety  feature  (going  from  3  to  2  safety  features) 

—  Although  we  will  not  be  discussing  this  in  the  interview,  value  analysis  of  the  two 
choices  continues  with  converting  range  of  measurement  to  value. 

—  It  is  determined  that  Safety  features  are  3  times  as  important  as  cost  (1  times  the 
entire  range  of  cost  =  1/3  entire  range  of  safety) 
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—  Combining  both  issues  to  compare  different  cars,  individual  weights  must  sum  to  1 

“cost’  +  ‘safety’  =  1 
3  x  ‘Cost’  =  ‘Safety’ 

Combining  two  equations,  ‘Cost’  +  (3  x  ‘Cost’)  =  1,  Therefore  Cost  =  0.25 

--  Weight  of ‘Cost’  is  0.25  and  weight  of ‘Safety’  is  0.75. 

-  Once  values  and  relative  weightings  identified  for  two  issues,  individual  cars  can  be 
compared  to  determine  ‘best  value’  car  for  the  buyer 

—  Comparison  of  different  cars  based  upon  summing  the  following  to  determine  total 
value: 

Value  of  ‘Cost’  x  Weight  of  ‘Cost’ 

Value  of  ‘Safety’  x  Weight  of  ‘Safety’ 

--  Using  example,  Car  ‘A’  commands  total  value  of: 

(0.5  x  0.25)  +  (1  x  0.75)  =  0.88 

—  Car  ‘B’  commands  total  value  of: 

(1.0  x  0.25)  +  (0.33  x  0.75)  =  0.50 

—  Therefore,  based  on  value  analysis,  choose  Car  ‘A’  as  car  with  highest  total  value 


3.  Over  the  course  of  this  research,  the  ranges  of  measurement  for  each  individual 
Campaign  Specific  Issue  has  been  determined.  The  relative  importance  ranking  of  the 
Campaign  Specific  Issues  have  also  been  identified.  The  purpose  of  the  interview  is  to 
obtain  the  relative  weightings  for  these  issues  in  the  manner  discussed  in  this  paper. 


4.  Thank  you  very  much  for  your  time  and  support  of  this  research  effort.  If  you  have 
any  questions  about  the  method  in  which  the  interview  will  be  conducted  or  for  any  other 
reason,  please  do  not  hesitate  to  e-mail  me  at  Paul.filcek@afit.af.mil. 


lLt  Paul  Filcek/AFIT/ENS/(937)254-5895/12Jan01 
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Appendix  G:  Delphi  Rounds  2  and  3  Questionnaire 


Please  answer  all  questions  on  this  document  and  return  within  an  e-mail  to: 

Paul.Filcek@afit.  af.mil 

The  Purpose  of  this  questionnaire  is  to  determine  relative  weightings:  between  the  sub¬ 
issues  within  a  parent  issue  (i.e.  Multi-National  Compatibility,  Allow  Assets  In,  etc. 
within  Host  Nation);  and  between  parent  issues  (i.e.  Host  Nation,  Enemy,  etc) 

During  previous  questionnaires,  the  relative  rankings  of  importance  between  the 
campaign  specific  issues  have  been  determined.  Based  upon  these  results,  issues 
recognized  to  be  more  potentially  influential  will  be  ranked  against  those  issues  not  as 
potentially  influential  to  the  selection  of  combat  aircraft  to  deploy  in  response  to  a 
contingency  crisis. 

Please  compare  the  Campaign  Specific  Issue  in  ‘Column  A’  to  the  Campaign  Specific 
Issue  in  ‘Column  B\  In  each  case,  the  issue  in  ‘Column  A’  has  been  identified  to  be 
more  potentially  important  than  the  issue  in  ‘Column  B’.  Looking  at  the  range  of 
measurement  for  each  issue,  PLEASE  STATE  HOW  MANY  TIMES  MORE 
IMPORTANT  THE  SWING  FROM  BEST  SCORE  TO  WORST  SCORE  FOR  THE 
ISSUE  IN  COLUMN  ‘A’  IS  THAN  THE  SWING  FROM  BEST  SCORE  TO  WORST 
SCORE  FOR  THE  ISSUE  IN  COLUMN  ‘B’. 

Consider  this  example.  Compare  Petroleum,  Oil  and  Lubricants  (POL)  to  Beddown 
Location:  Fighter  Distance.  The  swing  from  best  to  worst  score  for  POL  is  100  percent 
to  0  percent  support  of  asset  sorties  per  day.  The  swing  from  best  to  worst  score  for 
Beddown  Location:  Fighter  Distance  is  Omi  to  3000mi  distance  from  the  pre-determined 
staging  base  to  the  enemy  target.  I  believe  that,  if  POL  were  to  drop  from  100  percent  to 
0  percent,  this  drop  would  be  4  times  worse  than  if  Beddown  Location:  Fighter  Distance 
were  to  increase  from  0  mi  to  3000  mi  from  the  enemy  target.  As  my  answer,  I  put  4  in 
the  middle  block. 


COLUMN  A 

IS  HOW  TIMES  MORE 
IMPORTANT  THAN 

COLUMN  B 

Petroleum,  Oil,  Lubricants 
(100%  to  0%  asset  sets 
supported  per  day) 

4 

Beddown  Location:  Fighter 

Distance 

(0-3000mi  distance  to  enemy  target) 

Based  upon  this  example,  please  provide  your  answer  to  the  following  comparisons.  If 
you  feel  that  the  issue  in  ‘Column  A’  is  less  important  than  the  issue  in  ‘Column  B’, 
please  use  fractions  (1/2,  1/3,  etc.). 
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QUESTIONNAIRE  ANSWER  SHEET 


COLUMN  A 

IS  HOW  MANY 
TIMES  MORE 
IMPORTANT 
THAN 

COLUMN  B 

Petroleum,  Oil,  Lubricants 
(100%  to  0%  assets  supported/day) 

Beddown  Location:  Fighter  Distance 
(Omi  to  3000mi  distance  to  enemy) 

Enemy  Vs.  USAF  Capability 
(3  to  0  USAF  to  Enemy  Ratio) 

Petroleum,  Oil,  Lubricants 

(100%  to  0%  asset  sets  supported/day) 

Enemy  Vs.  USAF  Capability 
(3  to  0  USAF  to  Enemy  Ratio) 

Mission  Capable  Rate  of  Assets 
(100%  to  0%  average  MC  rate  of  acft) 

Mission  Capable  Rate  of  Assets 
(100%  to  0%  avg  MC  rate  of  acft) 

Multi-National  Compatibility 
(100%-0%  compatibility  w/  host) 

Beddown  Location:  Fighter  Distance 
(0-3000mi  distance  to  enemy  targets) 

Beddown  Location:  Bomber  Distance 
(0-7000mi  distance  to  enemy  target) 

Beddown  Location:  Fighter  Distance 
(0-3000mi  distance  to  enemy  target) 

Ability  to  Resupply 

(1  to  0  ratio:  consumption  to  resupply) 

Petroleum,  Oil,  Lubricants 
(100%  to  0%  assets  supported/day) 

Munitions  Storage  Area:  Fighter 
(100-0%  fighter  sorties  supported/day) 

Petroleum,  Oil,  Lubricants 
(100%  to  0%  assets  supported/day) 

Munitions  Storage  Area:  Bomber 
(100-0%  bomber  sorties  supported/day) 

Petroleum,  Oil,  Lubricants 
(100%  to  0%  assets  supported/day) 

Tanker  Support  Required 
(0-100%  sorties  supported/day) 

Topography 

(100%  to  0%  mission  degradation) 

Climate 

(100%  to  0%  mission  degradation) 

Mission  Capable  Rate  of  Assets 
(100%-0%  average  MC  rate  of  acft) 

Asset  Mission  Flexibility 
(100-0%  capability  of  all  aerospace 
missions) 

Mission  Capable  Rate  of  Assets 
(100%-0%  average  MC  rate  of  acft) 

Asset  Set  Utilization 

(0-100%  utility  of  all  aircraft  in  set) 

Enemy  Vs.  USAF  Capability 
(3  to  0  USAF  to  Enemy  Ratio) 

Enemy  Surveillance,  Intel,  and  Recon 
(0-100%  impact  of  exposure  to  enemy) 

Enemy  Vs.  USAF  Capability 
(3  to  0  USAF  to  Enemy  Ratio) 

Enemy  Interference 

(0%  to  100%  mission  degradation) 

Topography 

( 1 00%  to  0%  mission  degradation) 

Beddown  Location:  Fighter  Distance 
(Omi  to  3000mi  distance  to  enemy) 

Thank  you  very  much  for  your  support  of  this  research.  I  greatly  appreciate  the  time  you 
took  to  complete  these  questionnaires.  I  will  be  concluding  this  study  with  a  conclusion 
questionnaire  in  which  I  will  present  a  summary  of  these  individual  Campaign  Specific 
Issue  weights,  as  determined  by  the  expert  group.  You  will  have  an  opportunity  to  revise 
or  defend  you’re  your  weightings  in  this  final  questionnaire.  THANK  YOU  VERY 
MUCH! 
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Appendix  H:  USAF-Wide  Single-Dimension  Value  Functions 


USAF-Wide  SDVF: 
Multi-National  Compatibility 


1  0.8  0.6  0.4  0.2  0 

Range 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 

Percent 

Compatible 

Score 

1 

1 

0.4 

0.5 

Rho 

-25.00 

0 

0 

Rho 

15.00 

AEFC  SDVF 

Percent 

Compatible 

Score 

1 

1 

0.38 

0.5 

Rho 

-13.02 

0 

0 

Rho 

21.28 

SouthAF  SDVF 

Percent 

Compatible 

Score 

1 

1 

0.4 

0.5 

Rho 

-25.00 

0 

0 

Rho 

15.00 

CADRE  SDVF 

Percent 

Compatible 

Score 

1 

1 

0.84 

1 

0.35 

0.5 

Rho  -875 

0 

0 

Rho  -875 

ACC  SDVF 

Percent 

Compatible 

Score 

1 

1 

0.43 

0.5 

Rho 

-79.80 

0 

0 

Rho 

-60.20 
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USAF-Wide  SDVF: 

Beddown  Location-Fighter  Distance 


Computed  from  the  following  individual  single-dimension  value  functions: 


USAF-Wide  SDVF: 
Ability  to  Resupply 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 

Ratio:  Supply 
to  Consumption 

Score 

0 

0 

0.2 

0.15 

Rho  0.10 

0.6 

0.85 

1 

1 

Rho  -0.10 

SouthAF  SDVF 

Ratio:  Supply 
to  Consumption 

Score 

0 

0 

0.15 

0.15 

Rho 

150 

0.6 

0.7 

1 

1 

Rho 

-0.164 

CADRE  SDVF 

Ratio:  Supply 
to  Consumption 

Score 

0 

0 

0.35 

0.2 

Rho  0.042 

0.45 

0.55 

1 

1 

Rho  -0.0715 

AEFC  SDVF 

Ratio:  Supply 
to  Consumption 

Score 

0 

0 

0.65 

0.4 

Rho 

0.31 

1 

1 

Rho 

-0.18 

ACC  SDVF 

Ratio:  Supply 
to  Consumption 

Score 

0 

0 

0.5 

0.5 

Rho 

500 

1 

1 

Rho 

500 
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USAF-Wide  SDVF: 
MSA-Fighter 


Computed  from  the  following  individual  single-dimension  value  functions: 


|CentAF  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.31 

SouthAF  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.41 

CADRE  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

0.65 

0.5 

Rho  -0.1495 

1 

1 

Rho  0.26 

AEFC  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  1000 

ACC  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.56 
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USAF-Wide  SDVF: 
MSA-Bomber 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.31 

SouthAF  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.41 

CADRE  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

0.68 

0.5 

Rho  -0.2584 

1 

1 

Rho  0.28 

AEFC  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  1000 

ACC  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  -0.56 
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USAF-Wide  SDVF: 
POL 


061  Range  043 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 


Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  1000 

SouthAF  SDVF 

Percent  _ _ 

sorties  /  day  Score 


AEFC  SDVF 

Percent 

sorties  /  day 

Score 

0 

0 

1 

1 

Rho  1000 

|CADRE  SDVF 

Percent 

\wm 

sorties  /  day 

Score  | 

0 

0 

0.45 

0.5 

1 

1 

Rho  -0.5472 
Rho  0.19 


ACC  SDVF 

Percent 

sorties  /  day 

Score 

0 

1 

0 

1 

Rho 

1000 

6 


Computed  from  the  following  individual  single-dimension  value  functions 


SouthAF  SDVF 

Pet  Missions 

supported/day 

Score 

0 

1 

0 

1 

Rho 

1000 

CentAF  SDVF 

Pet  Missions 

supported/day 

Score 

0 

0 

1 

1 

Rho  1000 

CADRE  SDVF 

Pet  Missions 

supported/day 

Score 

0 

0 

1 

1 

Rho  1000 

AEFC  SDVF 

Pet  Missions 

supported/day 

Score 

0 

0 

1 

1 

Rho  1000 

ACC  SDVF 

Pet  Missions  _ 

supported/day  Score 


0 

1 


0 

1 


Rho  1000 


USAF-Wide  SDVF: 
Operating  Location-Topography 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 

Topography 

Difficulties 

Score 

0 

1 

1 

0 

Rho  -0.27 

SouthAF  SDVF 

Topography 

Difficulties 

Score 

0 

1 

1 

0 

Rho  -0.20 

CADRE  SDVF 

Topography 

Difficulties 

Score 

0 

1 

1 

0 

Rho  1.216 

AEFC  SDVF 

Topography 

Difficulties 

Score 

0 

1 

1 

0 

Rho  1000 

ACC  SDVF 

Topography 

Difficulties 

Score 

o  1 

1  0  Rho  -0.14 
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USAF-Wide  SDVF: 
Operating  Location-Climate 


Computed  from  the  following  individual  single-dimension  value  functions: 


CentAF  SDVF 

SouthAF  SDVF  | 

Climate 

Climate 

Difficulties 

Score 

Difficulties 

Score 

0 

1 

0 

1 

1 

0 

Rho  -0.27 

1 

0 

0  Rho  1000 


AEFC  SDVF 

Climate 

Difficulties 

Score 

0 

1 

1 

0 

Rho  1000 

CADRE  SDVF 

Climate 

Difficulties 

Score 

0 

i 

1 

0 

Rho  1.216 

ACC  SDVF 

Climate 

Difficulties 

Score 

0 

1 

1 

0 

Rho  -0.14 
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Computed  from  the  following  individual  single-dimension  value  functions: 


ISouthAF  SDVF 

Pet  Ops 

Exposure 

Score 

0 

1 

1 

0 

Rho  -0.66 

|CADRE  SDVF 

Pet  Ops 

Exposure 

Score 

0 

1 

0.56 

0.15 

Rho  -0.253 

1 

0 

CerrtAF  SDVF 

Pet  Ops 

Exposure 

Score 

0 

1 

1 

0 

Rho  -0.41 

AEFC  SDVF 

Pet  Ops 

Exposure 

Score 

0 

1 

1 

0 

Rho 

-0.49 

ACC  SDVF 

Pet  Ops 

Exposure 

Score 

o  1 

i  o 


Rho  1000 


USAF-Wide  SDVF: 
Enemy  Interference 


Computed  from  the  following  individual  single-dimension  value  functions: 


jCentAF  SDVF 

Enemy 

Effect 

Score 

0 

1 

0.5 

0 

Rho  -0.15 

i  o 


SouthAF  SDVF 

Enemy 

Effect 

Score 

0 

1 

1 

0 

Rho  -0.27 

CADRE  SDVF 

Enemy 

Effect 

Score 

0 

1 

0.8 

0 

Rho  -0.176 

1 

0 

AEFC  SDVF 

Enemy 

Effect 

Score 

0 

1 

0.3 

0.58 

Rho  0.42 

0.8 

0 

Rho  -0.36 

1 

0 

ACC  SDVF 

Enemy 

Effect 

Score 

0 

1 

0.36 

0.5 

Rho 

0.07 

1 

0 

Rho 

-0.12 

170 


Appendix  I:  Sensitivity  Analysis  on  Parent  Issue  Weights 


0.01  0.025  0.04  0.055  0.07  0.085  0.097  0.115  0.13  0.145 

Weights 


No  change  in  the  most  favored  force  mix  (#18)  within  the  range  of  weights  used. 


Just  one  change  in  the  most  favored  force  mix,  from  #19  to  #18,  within  the  range  of 
weights  used.  The  actual  change  in  rank  occurred  near  the  value  of  the  final  weight  used, 
lending  evidence  as  to  the  final  weight’s  appropriateness. 
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Mix  Value 


Sensitivity  Analysis:  Enemy  Considerations  Parent  Issue 


No  change  in  the  most  favored  force  mix  (#1 8)  within  the  range  of  weights  used 


Appendix  J:  Sensitivity  Analysis  on  the  USAF-Wide  SDVF’s 


Sensitivity  Analysis:  USAF-Wide  Single-Dimension  Value 

Function 


Each  of  the  16  USAF-wide  SDVF’s  was  constructed  from  the  80  individual 
SDVF’s  elicited  from  AEFC,  CentAF,  ACC,  CADRE,  and  SouthAF  (five  for  each  of  16 
factors).  The  16  USAF-wide  SDVF’s  comprise  the  extrinsic  force  evaluation  tool  (go/no 
go  filter  notwithstanding). 

The  graph  above  represents  the  top  four  force  mixes  as  scored  using  the  USAF- 
wide  SDVF’s.  However,  each  point  along  the  X-axis  has  one  of  the  16  factors 
represented  by  the  most  divergent  individual  SDVF for  that  factor  (i.e.,  AEFC  for  POL). 

As  can  be  seen,  none  of  the  top  four  force  mixes  change  their  relative  ranking 
using  a  single  dissenting  SDVF.  This  provides  evidence  for  the  robustness  of  the  USAF- 
wide  SDVF’s. 

It  is  important  to  note  that  a  small  number  of  changes  in  rankings  did  occur 
among  lower  ranked  force  mixes.  However,  these  changes  were,  in  every  case,  among 
force  mixes  of  almost  identical  assets.  Among  differentiated  force  mixes,  no  rank 
changes  occurred. 
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Appendix  K:  Input  Sheets  for  Scoring  Force  Mixes 


The  “Go  /No  Go  Filter”  (constraints)  sheet,  less  the  Go  / No  Go  score  column  (included 
in  actual  sheet).  The  constraints  are  specific  to  the  aircraft  types  within  a  candidate  force 
mix,  which  then  drive  a  force  mix’s  initial  feasibility.  This  sheet  is  linked  by  formulas  to 
the  “Scored  Force  Mixes”  sheet.  Manual  inputs  into  either  sheet  affect  the  other. 


FA  FB  FC  B1  B2 


!  Test  1  {POLITICS:  Force  Mix  Allowed? .  11  1  1  1 

J _  FLAG: 


Test  2 {OPERATING LOCATION:  Runway  Length/Width  ‘""'"T" '  11  1  1 

j _ _ _  FLAG: 

Test  Z  {OPERATING  LOCATION:  Runway  Weight  Capacity  1  11  1  1 

1  _  FLAG: 

Test  4 {OPERATING  LOCATION:  Ramp  Space  1  111  1 

1  _ ______ _ ___ _ ■  FLAG:  F 


Tests  jOesired  Force  Mix  AVAILABLE?  . . . i  1 "  1  I  t 

{(reconstitution,  rotations,  fleet  modifications,  etc)  FLAG:  F 


Test  8  jNON-iFR  DISTANCE:  <700tT*/Figbter,  3000n*/Bomber  1  11  11 

{(applies  If  no  tankers  available,  distance  can't  exceed)  flag: 


*  *  {  F;  A  condition  that  must  be  accounted  for  in  the  constraint  formula  specific  to  the  fo  rce 


Calc  box  for  distance: 


Tankers  Available:  1 

Miles  from  target/F:  0 

Miles  from  target/B:  1 
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The  “Straw  Model”  sheet,  including  situational  (host  nation/base)  inputs  but  less  the 
breakout  of  force  mix  composition  and  suitability  (in  the  actual  spreadsheet).  This  sheet 
is  the  input  for  the  “Scored  Force  Mixes”  formula  sheet  as  well  as  the  “Go  / No  Go”  filter 
sheet. 


HOST  NATION:  Republic  of  Cantonia 

STAGING  BASE:  Blue  Coral  AB,  ROC 


STATUS 

FA 

FB 

FC 

B  1 

B2 

MULTI-NATIONAL  COMPATIBILITY: 

No  Mult. 

0.7 

0.9 

0.8 

0.9 

0.7 

DISTANCE  FROM  TARGETS: 

900 

0.9 

1 

1.1 

2.5 

2.7 

MUNITIONS  STORAGE  CAPACITY: 

0.95 

0.95 

0.75 

0.85 

0.7 

0.8 

POL  CAPACITY: 

0.9 

0.8 

0.9 

0.75 

0.8 

0.7 

TANKER  AVAILABILITY: 

0.9 

TOPOGRAPHY  IMPACT: 

0.2 

CLIMATE  IMPACT: 

0.22 

ENEMY  CAPABILITY: 

33 

ENEMY  ISR  MULTIPLIER: 

0.4 

0.15 

0.2 

0.25 

0.22 

0,11 

ENEMY  INTERFERENCE  IMPACT: 

0.3 

90-DAY  RUNNING  MC  RATE  OF  MDS: 

No  Mult. 

0.92 

0.83 

0.8 

0.75 

0.84 

*  BLUE  PRINT:  SPECIFIC  TO  HOST  BASE 

(GUI  idea:  user  can  scroll  a  list  of  MDS's,  gain  90-day  MC  rate  and  particular  Multi-National  Compatibility  score-Perfect  for  ALP's  real  time  info  networ 
ALP's  real  time  logistics  architecture  can  have  ALL  of  the  BLUE  data  for  host  bases  continually  updated,  to  be  displayed  and  calculated  within  model 
automatically  upon  base  selection. 


Multi-  Beddown  Beddown  MSA  MSA  Tanker  Enemy  Enemy  Enemy 

National  Fighter  Bomber  Resupply  Fighter  Bomber  EOL  Flexibility  Utilization  Support  Topography  Climate  MC  Bate  Capability  ISR  Interference 


0.84 

0.87 

2.25 

0.83 

0.78 

0.66 

0.78 

0.83 

0.32 

0.70 

0.16 

0.16 

0.85 

1.18 

0.07 

0.24 

0.78 

0.84 

2.25 

0.77 

0.84 

0.66 

0.75 

0.79 

029 

0.67 

0.17 

0.15 

0.88 

1.18 

0.07 

0.26 

0.76 

0.84 

2.43 

0.77 

0.84 

0.76 

0.74 

0.78 

0.28 

0.68 

0.17 

0.16 

0.89 

1.18 

0.06 

0.26 

0.82 

0.87 

2.43 

0.84 

0.78 

0.76 

0.77 

0.82 

0.31 

0.70 

0  16 

0.16 

0.86 

1.18 

0.07 

0.24 

0.78 

0.93 

2.25 

0.77 

0.84 

0.66 

0.69 

0.72 

027 

0.76 

0.17 

0.18 

0.83 

1.18 

0.09 

0.22 

0.84 

0.96 

2.25 

0.83 

0.78 

0.66 

0.72 

0.76 

0.31 

0.78 

0.16 

0.19 

0.81 

1.18 

0.09 

021 

0.76 

0.93 

2.43 

0.77 

0.84 

0.76 

0.69 

0.71 

027 

0.76 

0.17 

0.18 

0.84 

1.18 

0.08 

022 

0.82 

0.96 

2.43 

0.84 

0.78 

0.76 

0.71 

0.74 

0.30 

0.79 

0.17 

0.19 

0.81 

1.18 

0.09 

021 

0.87 

0.93 

225 

0.86 

0.74 

0.66 

0.76 

0.81 

0.33 

0.76 

0.16 

0.18 

0.81 

1.18 

0.09 

0.22 

0.75 

0.87 

225 

0.73 

0.87 

0.66 

0.71 

0.74 

0.27 

0.70 

0.18 

0.16 

0.87 

1.18 

0.07 

0.24 

0.85 

0.93 

2.43 

0.87 

0.74 

0.76 

0.75 

0.80 

0.32 

0.76 

0.16 

0.18 

0.82 

1.18 

0.08 

0.22 

0.73 

0.87 

2.43 

0.74 

0.87 

0.76 

0.70 

0.73 

0.26 

0.70 

0.18 

0.16 

0.88 

1.18 

0.07 

0.25 

0.83 

0.87 

2.31 

0.83 

0.78 

0.70 

0.77 

0.83 

0.32 

0.70 

0.16 

0.16 

0.85 

1.18 

0.07 

0.24 

0.77 

0.84 

2.37 

0.77 

0.84 

0.73 

0.74 

0.79 

0.28 

0.68 

0.17 

0.15 

0.88 

1.18 

0.07 

0.26 

0.77 

0.84 

2.31 

0.77 

0.84 

0.70 

0.75 

0.79 

029 

0.67 

0.17 

0.15 

0.88 

1.18 

0.07 

0.26 

0.83 

0.87 

2.37 

0.83 

0.78 

0.73 

0.77 

0.82 

0.31 

0.70 

0.16 

0.16 

0.86 

1.18 

0.07 

0.24 

0.77 

0.93 

2.31 

0.77 

0.84 

0.70 

0.69 

0.72 

0.27 

0.76 

0.17 

0.18 

0.84 
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“Input  Sheet”  for  the  Straw  Model: 

*  Based  upon  notional  suitability  ratings  for  required  missions  AA,  AG,  and  PB,  the  five 
available  aircraft  are  mixed  into  candidate  forces  that  meet  the  mandatory  mission 
capability  for  day  X.  Each  force  mix  then  produces  its  own  suitability  score.  The 
combination  of  suitability  and  capability  comprise  an  intrinsic  value.  The  suitability 
scores  and  rankings  of  scores  in  the  right  hand  columns  are  not  to  be  confused  with 
overall  intrinsic  values. 
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Appendix  L:  The  Subject  Matter  Expert  Pool 


Col.  Joel  Petersen 

Lt.  Col.  Larry  Brockshus 

Mr.  Gary  Hitz 

Mr.  William  Mattson 

Lt.  Col.  Connie  Vandermarliere 

Lt.  Col.  Michael  Duvall 

Lt.  Col.  Petitto  McMahon 

Maj.  Dennis  McDevitt 

Maj.  William  Carter 

Maj.  Gregory  Broardt 

Maj.  Lou  Kaelin 

Lt.  Col.  William  Doneth 

Maj.  Thomas  Nankervis 

Mr.  Jeffrey  Brown 

Mr.  Don  Niederkopfler 

Lt.  Col.  Barrett  Clay 

Lt.  Col.  N.  Gonzalez 

Maj.  Steven  Wackowski 

Col.  Kenneth  Poole 

Lt.  Col.  Bryan  Johnson 

Maj.  Kirby  Locklear 

Mr.  Clay  Olschner 

Mr.  Michael  Urban 

Maj.  Chuck  McCleen 

Lt.  Col.  Marty  Spitek 

Col.  Ernest  Howard 

Col.  Shirley  Rawls 

Lt.  Col.  Raymond  Santiago 

Maj.  Laletta  Tatum 

Mr.  Robert  Barthelmess 

Dr.  Adolph  Perry 

Maj.  J.  D.  Haas 

Lt.  Col.  John  Stankowski,  III 

Lt.  Col.  Burrell  Jones,  III 
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Appendix  M:  Visual  Basic  Coding  for  SDVF’s 


The  following  code  for  Microsoft  Excel  Visual  Basic  was  provided  by  Kirkwood 
for  converting  hand-drawn  value  function  elicitations  into  exponential  or  piecewise  linear 
value  function  curves  [Kirkwood,  1997:  81]. 


Sub  kirkwood  (expon) 

Function  ValueE(x,  low,  high,  monotonicity,  rho) 

'  Determines  the  value  score  for  an  expon  value  function 

Select  Case  UCase(monotonicity) 

Case  "I" 

delta  =  x  -  low 
Case  "D" 

delta  =  high  -  x 
End  Select 

If  UCase(rho)  =  "INF"  Then 

ValueE  =  delta  /  (high  -  low) 

Else 

ValueE  =  (1  -  Exp(-delta  /  rho))  /  (1  -  Exp(-(high  -  low)  /  rho)) 

End  If 

End  Function 


Function  ValuePL(x,  Xi,  Vi) 

'  Determines  the  value  score  for  a  piecewise  linear  function 
i  =  2 

Do  While  x  >  Xi(i) 

i  =  i  +  1 

Loop 

ValuePL  =  Vi(i  - 1)  +  (Vi(i)  -  Vi(i  - 1))  *  _ 

(x  -  Xi(i  - 1))  /  (Xi(i)  -  Xi(i  - 1)) 

End  Function 
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Appendix  N:  Excel  Formulas  Used  to  Incorporate  the  16  SDVF’s 


*The  first  two  formulas  provide  short  explanations.  The  MSA  factors  for  Bombers  and 
Fighters,  as  well  as  the  Distance  factors  for  both,  are  complementary  in  their  weighting. 
The  short  explanation  provided  for  “Beddown  Location — Fighter  Distance”  (second, 
below)  also  pertains  to  “MSA — Fighter,”  “MSA — Bomber,”  and  “Beddown  Location — 
Bomber.” 

Excel  Formula  for  scoring  the  “Multi-National  Compatibility”  SDVF  vs.  a  candidate 
force  mix: 

=$D$4*IF('Constraint  Filter'!$AA3>0,IF('Straw  Model'!  V4>'Multi-National 
Compatibility' ! $ AE$  1 1 ,1 , IFf  Straw  Model'!  V4>'Multi-National 
Compatibility'!$AE$12,0  5+0  5*ValueE('Straw  Model'!V4, 'Multi-National 
Compatibility' !$ AES  12, 'Multi-National  Compatibility' !$AE$1 1 , "I", 'Multi- 
National  Compatibility' !  $  AH$  1 2),0.5  *  ValueE('Straw  Model' !  V4, 'Multi-National 
Compatibility’ !  $  AES  1 3 , 'Multi-National  Compatibility' !  $  AES  1 2,  "I", 'Multi- 
National  Compatibility'!  $AH$  1 3))),0) 

Where  the  first  shaded  area  is  the  global  weight,  the  second  shaded  area  is  the  go  / 
no  go  feasibility  multiplier.  The  IF  statements  compute  different  segments  along 
the  value  function’s  curve,  and  the  shaded  multipliers  maintain  value 
advancement  along  the  curve. 


Excel  Formula  for  scoring  the  “Beddown  Location — Fighter  Distance”  SDVF  vs.  a 
candidate  force  mix: 

=(($E$4*(SUM('Straw  Model' !P4:R4)/(’Straw  Model' !04)))+('Scored  Force 
Mixes' ! $F$4-((SUM('Straw  Model’! S4:T4))/'Straw  Model'!04)*'Scored  Force 
Mixes' !  $F$4))*IF  ('Constraint  Filter' !  $  AA3>0,IF('Straw  Model' !  W4<’Beddown 
Fighter' !$AJ$1 1,1, IF('Straw  Model'!W4<Beddown 

Fighter' !$AJ$1 2,  ValueE('Straw  Model' !W4, 'Beddown  Fighter’  !$AJ$1 1, 'Beddown 
Fighter' !$AJ$1 2, "D", 'Beddown  Fighter'!  SAMS  1 2), 0)),0) 

Where  the  first  shaded  area  represents  a  complementary  relationship  between  this 
factor’s  weight  and  that  of  “Bomber  Distance.”  It  is  this  factor’s  weight  times  its 
percent  of  the  force  mix  plus  the  sum-product  of  the  other  factor’s  weight  minus 
its  percentage  of  the  force  mix  times  its  weight.  Other  components  of  the  formula 
are  similar  to  the  “multi-National  Compatibility”  factor’s  formula. 
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Excel  Formula  for  scoring  the  “Beddown  Location — Bomber  Distance”  SDVF  vs.  a 
candidate  force  mix: 

=(($F$4*(SUM('Straw  Model' !S4:T4))/'Straw  Model' !04)+($E$4-((SUM('Straw 
Model' !P4:R4))/('Straw  Model'!04))*('Scored  Force 

Mixes'!$E$4)))*(IF('Constraint  Filter'!  AA3>0,IF('Straw  Model' !X4<’Beddown 
Bomber'  !$AJ$1 1,1,  IF('Straw  Model'!X4<’Beddown  Bomber'!$AJ$12, 
ValueE('Straw  Model' !X4, 'Beddown  Bomber'!$AJ$l  1, 'Beddown 
Bomber' !  $  A  J$  1 2,  ”D", 'Beddown  Bomber' !  $  AMS  1 2),0)),0)) 


Excel  Formula  for  scoring  the  “Resupply”  SDVF  vs.  a  candidate  force  mix: 

=$G$4*IF('Constraint  Filter' !$AA3>0,IF('Straw  Model' !Y4<’ Ability  to 
Resupply'!$AJ$l  l,0.18*ValueE('Straw  Model' !Y4,' Ability  to 
Resupply'!$AJ$l  1, 'Ability  to  Resupply'!$AJ$10,"D", 'Ability  to 
Resupply'lSAMSl  l),IF('Straw  Model’ !Y4<’Ability  to 
Resupply'!$AJ$12,ValuePL('Straw  Model' !Y4,' Ability  to 
Resupply' !$AJ$1 1  :$AJ$12, 'Ability  to 

Resupply' !  $  AK$  1 1  :$AK$1 2), 0.62+0.3  8*  ValueE(’Straw  Model' !Y4, 'Ability  to 
Resupply'!$AJ$13, 'Ability  to  Resupply'!$AJ$12,"D", 'Ability  to 
Resupply' !  $  AM$  1 3  ))),0) 


Excel  Formula  for  scoring  the  “MSA — Fighter”  SDVF  vs.  a  candidate  force  mix: 

=(($H$4*((SUM('Straw  Model'!P4:R4))/('Straw  Model’ !04)))+((’Scored  Force 
Mixes'!  $I$4)-(((SUM('Straw  Model’!  S4:T4))/('Straw  Model'!04))*('Scored  Force 
Mixes'!$I$4))))*(IF('Constraint  Filter'!$AA3>0,IF('Straw 
Model'!  Z4<MSA_Fighter!$AJ$l  1 ,0.5*ValueE('Straw 

Model' !  Z4,MS  A_Fighter !  $  A  J$  1 0,MS  A_Fighter !  $  AJ$  1 1  ,"I",MS  A_Fighter !  $  AM$ 

1 1) , 0.5+0. 5*ValueE('Straw 

Moder!Z4,MSA_Fighter!$AJ$ll,MSA_Fighter!$AJ$12,"I",MSA_Fighter!$AM$ 

12) ), 0)) 
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Excel  Formula  for  scoring  the  “MSA — Bomber”  SDVF  vs.  a  candidate  force  mix: 


=(($I$4*((SUM('Straw  Model'!S4:T4))/('Straw  Model' !04)))+((’Scored  Force 

Mixes' !$H$4)-(((SUM('Straw  Moder!P4:R4))/('Straw  Model'!  04))*('Scored 

Force  Mixes'!$H$4))))*(EF('Constraint  Filter'!$AA3>0,IF('Straw 

Model'!  AA4<MSA_Bomber!$AJ$l  1,0. 5*ValueE('Straw 

Model'!  AA4,  MS  A_Bomber !  $  AJ$  1 0,MS  A_Bomber !  $  AJ$  1 1  ,"I"  ,MS  A_Bomber !  $ 

AM$1 1), 0.5+0. 5*ValueE('Straw 

Model'!AA4,MSA_Bomber!$AJ$ll,MSA_Bomber!$AJ$12,’T’,MSA_Bomber!$ 

AM$12)),0)) 


Excel  Formula  for  scoring  the  “POL”  SDVF  vs.  a  candidate  force  mix: 

=$J$4*IF('Constraint  Filter'!$AA3>0,IF('Straw 
Model'!  AB4<POL!$AJ$  1 1 ,0.5*ValueE(’Straw 

Model' !  AB4,POL!  $  AJ  $  1 0,POL!  $  AJ$  1 1  ,"F',POL !  $AM$  1 1  ),0. 5+0.5  *V  alueE('Stra 
w  Model' !  AB4,POL!$AJ$  1 1  ,POL!  $AJ$  1 2,"I",POL!  $  AM$  1 2)),0) 


Excel  Formula  for  scoring  the  “Inter-Mission  Flexibility”  SDVF  vs.  a  candidate  force 
mix: 


=$K$4*IF('Constraint  Filter'!$AA3>0,IF('Straw 
Model' !  AC4<Flexibility !  $  A  J$  1 1 ,  V  alueE('Straw 

Model’ !  AC4,  Flexibility !  $  AJ$  1 0, Flexibility !  $  AJ$  1 1  ,"I",  Flexibility!  $  AM$  1 1 ),  1  ),0) 


Excel  Formula  for  scoring  the  “Force  Mix  Utilization”  SDVF  vs.  a  candidate  force  mix: 

=$L$4*IF('Constraint  Filter’  !$AA3>0,IF('Straw 
Model' !  AD4<Utilization!  $  A  J  $  1 1 , 1  ,IF  ('Straw 
Model' !  AD4<Utilization!  $  AJ  $  1 2,  V  aluePL('Straw 

Model'!  AD4,Utilization!$AJ$l  1  :$AJ$12,Utilization!$AK$l  1  :$AK$12),0.94*Val 
ueE('Straw 

Model' !  AD4, Utilization !  $  AJ$  1 2, Utilization!  $AJ$  1 3, "D", Utilization!  $AM$  1 3))), 

0) 
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Excel  Formula  for  scoring  the  “Tanker  Support  Available”  SDVF  vs.  a  candidate  force 
mix: 


=$M$4*IF('Constraint  Filter'  !$AA3>0,IF('Straw  Model'!AE4<'Tanker 
Support' !$  AJ$  1  l,ValueE('Straw  Model'!AE4, 'Tanker  Support' !$AJ$  10,’ 'Tanker 
Support' !$  AJ$  1 1, "I”, 'Tanker  Support' !$AM$1 1),1),0) 


Excel  Formula  for  scoring  the  “Operating  Location — Topography”  SDVF  vs.  a  candidate 
force  mix: 

=$N$4*IF('Constraint  Filter'!$AA3>0,IF('Straw 
Model'!  AF4<Topography!  $AJ$  1 1  ,ValueE('Straw 

Model'!  AF4, Topography!  $  AJ$  1 0, Topography!  $  AJ$  1 1 , "D", Topography  !$  AMS  1 1 

),0),0) 


Excel  Formula  for  scoring  the  “Operating  Location — Climate”  SDVF  vs.  a  candidate 
force  mix: 

=$0$4*IF('Constraint  Filter' !  $AA3>0,IF('Straw 
Model' !  AG4<Climate !  $  AJ$  1 1 ,  V  alueE('Straw 

Model' !  AG4, Climate!  $AJ$  1 0, Climate !  $  A  J$  1 1  ,"D", Climate !  SAMS  1 1  ),0),0) 


Excel  Formula  for  scoring  the  “Force  Mix  MC  Rate”  SDVF  vs.  a  candidate  force  mix: 

=$P$4*  IF(’Constraint  F  i  Iter' !  $  AA3  >0,IF  ('Straw 
Model' !  AH4<  Availability!  $  A  J$  1 1 ,0,IF  ('Straw 
Model'!  AH4<Availability!$AJ$12,0.4*ValueE('Straw 

Model'!AH4,Availability!$AJ$l  l,Availability!$AJ$12,"I",Availability!$AM$12), 
IF('Straw  Model'!AH4<Availability!$AJ$13,0.4+0.6*ValueE('Straw 
Moder!AH4,Availability!$AJ$12,Availability!$AJ$13,"I",Availability!$AM$13), 
1))),0) 
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Excel  Formula  for  scoring  the  “Enemy  Capability”  SDVF  vs.  a  candidate  force  mix: 

=$Q$4*IF('Constraint  Filter' !$AA3>0,IF(lStraw  Model'!AI4<'Enemy 
Capability'!  $  AJ$  1  l,0,IF('Straw  Model' !AI4<’Enemy 
Capability' !  $  A  J$  1 2,0.42*  V  alueE('Straw  Model' !  AI4, 'Enemy 
Capability'!$AJ$l  1, 'Enemy  Capability' !$AJ$12,"r','Enemy 
Capability' !$  AMS  1 2),IF('Straw  Model'  !AI4<’Enemy 
Capability' !  $  AJ$  1 3 ,0.42+0. 5  8 * V  alueE('Straw  Model' !  AI4, 'Enemy 
Capability'!$AJ$12,'Enemy  Capability'!$AJ$13,"I",'Enemy 
Capability'!  $AM$  1 3),  1))),0) 


Excel  Formula  for  scoring  the  “Enemy  ISR”  SDVF  vs.  a  candidate  force  mix: 

=$R$4*IF('Constraint  Filter' !$AA3>0,IF('Straw  Model’!AJ4<’Enemy 
ISR'!$AJ$1  l,ValueE('Straw  Model'!AJ4, 'Enemy  ISR'!$AJ$10, 'Enemy 
ISR’JSAJSl  1,"D", 'Enemy  ISR'!$AM$1 1),0),0) 


Excel  Formula  for  scoring  the  “Enemy  Interference”  SDVF  vs.  a  candidate  force  mix: 

=$S$4*IF('Constraint  Filter' !$AA3>0,IF('Straw  Model' !AK4<’Enemy 
Interference'!  SAJS 1 1 ,0. 5+0. 5  *V  alueE('Straw  Model’!AK4, 'Enemy 
Interference'!$AJ$10,'Enemy  Interference' !$AJ$1 1,"D",' 'Enemy 
Interference'!$AM$l  l),IF('Straw  Model' !AK4<’Enemy 
Interference' !  $  AJS 1 2,0. 5  *  ValueE('Straw  Model' !  AK4,'Enemy 
Interference'!  $AJ$  1 1,1 'Enemy  Interference' !$AJ$12,"D", 'Enemy 
Interference' !  SAMS  1 2) ,  0)),0) 
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Appendix  O:  External  Sensitivity  Analyses 


(suitability)  _  _  (suitability) 
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Intrin  @  1 

Multiplic. 

MIX 

Extrin  @  | 

Intrin  @ 

.9  and  .1 

1 

Multiplic.  | 

Typical 

Rank 

Typical 

Rank 

Result 

Rank  j 

Typical 

Rank 

Rank 

Result 

Rank  | 

1 

0.713534 

15 

25.6 

1 

18.26647 

1 

1 

0.713534 

15 

28.8 

1 

20.54978 

1 

2 

0.714978 

8 

12.6 

48 

9.008728 

48 

2 

0.714978 

8 

4.8 

44 

3.431896 

44 

3 

0.71603 

2 

11.8 

51 

8.449152 

51 

3 

0.71603 

2 

3.2 

50 

2.291295 

50 

4 

0.714309 

9 

24.8 

4 

17.71487 

4 

4 

0.714309 

9 

27.2 

7 

19.42921 

7 

5 

0.710151 

34 

21.6 

9 

15.33927 

9 

5 

0.710151 

34 

12.8 

V  21  ' 

9.089935 

23 

6 

0.70924 

38 

16.6 

39 

11.77338 

39 

6 

0.70924 

38 

4.8 

44 

3.404351 1 

45 

7 

0.711153 

25 

20.8 

14 

14.79197 

15 

7 

0.711153 

25 

11.2 

«•  37 

7.964909 

42 

8 

0.709751 

35 

15.8 

43 

11.21406 

43 

8 

0.709751 

35 

3.2 

50 

2.271203 

51 

9 

0.710476 

30 

23.6 

5 

16.76722 

5 

9 

0.710476 

30 

28.8 

1 

20.4617 

2 

10 

0.712788 

16 

15.6 

44 

11.11949 

44 

10 

0.712788 

16 

12.8 

21 

9.123686 

22 

11 

0.711142 

26 

22.8 

8 

16.21403 

8 

11 

0.711142 

26 

27.2 

7 

19.34305 

8 

12 

0.714035 

10 

14.8 

47 

10.56772 

47 

12 

0.714035 

10 

11.2 

37 

7.997193 

38 

13 

0.713743 

12 

25.2 

2 

17.98633 

3 

13 

0.713743 

12 

28 

3 

19.98481 

4 

14 

0.715651 

4 

12.2 

49 

8.730945 

49 

14 

0.715651 

4 

4 

46 

2.862605 

46 

15 

0.715283 

5 

12.2 

49 

8.726451 

50 

15 

0.715283 

5 

4 

46 

2.861131 

47 

16 

0.71402 

11 

25.2 

2 

17.9933 

2 

16 

0.71402 

11 

28 

3 

19.99255 

3 

17 

0.710439 

31 

21.2 

11 

15.06131 

11 

17 

0.710439 

31 

12 

28 

8.525268 

31 

18 

0.709478 

36 

16.2 

41 

1 1 .49355 

41 

18 

0.709478 

36 

4 

46 

2.837913 

48 

19 

0.710439 

31 

21.2 

11 

15.06131 

11 

19 

0.710439 f' 

31 

12 

28 

8.525268 

31 

20 

0.709478 

36 

16.2 

41 

1 1 .49355 

41 

20 

0.709478 

36 

4 

46 

2.837913  L 

48 

21 

0.710648 

27 

23.2 

6 

16.48703 

6 

21 

0.710648 

27 

28 

3 

19.89813  [ 

5 

22 

0.713594 

13 

15.2 

45 

10.84662 

45 

22 

0.713594 

13 

12 

&  28 

8.563124 

29 

23 

0.710648 

27 

23.2 

6 

16.48703 

6 

23 

0.71 0648  N 

27 

28 

f  3 

1 9.8981 3  H 

5 

24 

0.713594 

13 

15.2 

45 

10.84662 

45 

24 

0.71 3594  f 

13 

12 

28 

8.563124 

29 

25 

0.715222 

6 

17.6 

34 

12.58791 

34 

25 

0.715222 

6 

12.8 

fr  21 

9.154844?; 

21 

26 

0.716797 

1 

16.8 

37 

12.04219; 

36 

26 

0.716797 

1 

11.2 

IV  37 

8.028124 

37 

27 

0.715707 

3 

17.2 

35 

12.31016 

35 

27 

0.715707  ' 

3 

12 

:  28 

8.58848 

28 

28 

0.711992 

23 

20.6 

16 

14.66704 

16 

28 

0.711992 

23 

20.8 

SV.9 

14.80944 

9 

29 

0.712621 

20 

19.8 

19 

14.10989 

18 

*\v  "■■■..= 

30 

0.712379 

22 

20.2 

17 

14.39005 

17 

29 

0.712621 

20 

19.2 

V  13  ; 

13.68232 

14 

31 

0.70073 

50 

1 9.6 

21 

13.7343 

21 

30 

0.712379 

22 

20 

ii  : 

14.24757  civil  : 

32 

0.710606 

29 

18.8 

26 

13.3594 

26 

31 

0.70073  ■ 

50 

12.8 

r.  21 

8.969341 

26 

33 

0.70368 

45 

19.2 

23 

13.51066 

23 

32 

0.710606 

29 

11.2 

S 

7.958791 

43 

34 

0.71145 

24 

18.7 

27 

13.30412 

27 

33 

0.70368 

45 

12 

28 

8.444161  ii,- 

36 

35 

0.704337 

44 

19.7 

20 

13.87544 

20 

34 

0.71145 ! 

24 

16 

17 

.  ■ 

11.3832 

17  \ 

36 

0.708184 

40 

18.2 

32 

12.88894 

32 

35 

0.704337 

44 

16 

17 

1 1.2694 1;; 

18 

37 

0.704945 

43 

19.1 

24 

13.46444 

24 

36 

0.708184^ 

40 

12 

28 

8.498204 

34 

38 

0.701729 

49 

20.1 

18 

14.10476 

19 

37 

0.704945 

43 

16.8 

£:!■:  15  ■ 

11.84307 

15 

39 

0.700554 

51 

18.6 

29 

13.03031 

31 

38 

0.701729 

49 

16.8 

15 

11.78905 

16 

40 

0.71521 

7 

18.3 

30 

13.08835 

29 

39 

0.700554 

51 

12.8 

21 

8.967096 

27  ; 

41 

0.710439 

33 

19.3 

22 

13.71147 

22 

40 

0.71521; 

7 

15.2 

19 

10.8712 

19 

42 

0.712593 

21 

17.8 

33 

12.68415 

33 

41 

0.710439 

33 

15.2 

19 

10.79867 

20 

43 

0.705602 

41 

18.7 

27 

13.19475 

28 

42 

0.712593 

21 

11.2 

37 

7.981041 

41 

44 

0.70902 

39 

16.7 

38 

1 1 .84063 

38 

43 

0.705602 

41 

12 

28 

8.46722 

35 

45 

0.705602 

41 

21.2 

11 

14.95876 

13 

44 

0.70902  - 

39 

12 

:  28 

8.508238 

33 

46 

0.702531 

46 

19.1 

24 

13.41834 

25 

45 

0.705602 

41 

20 

11 

14.11203 

12 

47 

0.702531 

46 

17.1 

36 

12.01328 

37 

46 

0.702531 

46 

12.8 

21 

8.992394 

24 

48 

0.702531 

46 

21.6 

9 

15.17466 

10 

47 

0.702531 

46 

12.8 

21 

8.992394 

24 

49 

0.712772 

17 

18.3 

30 

13.04373 

30 

48 

0.702531 

46 

20.8 

9 

14.61264 

10 

50 

0.712772 

17 

16.3 

40 

11.61819 

40 

49 

0.712772 

17 

11.2 

37 

7.983048 

39 

51 

0.712772 

17 

20.8 

14 

14.82566 

14 

50 

0.712772 

17 

11.2 

37 

7.983048 

39 

51 

0.712772 

17 

19.2 

13 

13.68523 

13 
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(suitability) 

(suitability) 

MIX 

Extrin  @  | 

Intrin  @ 

Muitiplic.  | 

MIX 

Extrin  @  | 

Intrin  (S)  | 

Muitiplic.  [ 

Typical 

Rank 

.8  and  .2 

Rank 

Result 

Rank  | 

Typical  Rank 

.7  and  .3 

Rank 

Result 

Rank 

1 

0.713534 

15 

25.6 

1 

18.26647 

1 

1 

0.713534 

15 

22.4 

1 

15.98316 

1 

2 

0.714978 

8 

7.6 

44 

5.433836 

44 

2 

0.714978 

8 

10.4 

44 

7.435775 : 

44 

3 

0.71603 

2 

6.4 

50 

4.582591 

50 

3 

0.71603 

2 

9.6 

50 

6.873886 

50 

4 

0.714309 

9 

24.4 

7 

17.42915 

7 

4 

0.714309 

9 

21.6 

7 

15.42908 

7 

5 

0.710151 

34 

13.6 

21 

9.658056 

23 

5 

0.710151 1 

34 

14.4 

21 

10.22618 

23 

6 

0.70924 

38 

7.6 

44 

5.390222 

45 

6 

0.70924 

38 

10.4 

44 

7.376093 

45 

7 

0.711153 

25 

12.4 

37 

8.818292 

42 

7 

0.711153 

25 

13.6 

37 

9.671675!^ 

42 

8 

0.709751 

35 

6.4 

50 

4.542406 

51 

8 

0.709751 

35 

9.6 

50 

6.813609 

51 

9 

0.710476 

30 

25.6 

1 

18.18817 

2 

9 

0.710476 

30 

22.4 

1 

15.91465 

2 

10 

0.712788 

16 

13.6 

21 

9.693916 

22 

10 

0.712788 

16 

14.4 

21 

10.26415; 

22 

11 

0.711142 

26 

24.4 

7 

17.35186 

8 

11 

0.711142 

26 

21.6 

7 

15.36066; 

8 

12 

0.714035 

10 

12.4 

37 

8.854035 

38 

12 

0.714035 

10 

13.6 

37 

9.710877 

38 

13 

0.713743 

12 

25 

3 

17.84358 

4 

13 

0.713743 

12 

22 

3 

1 5.70235 : 

4 

14 

0.715651 

4 

7 

46 

5.009559 

46 

14 

0.715651 

4 

10 

46 

7,156512 

46 

15 

0.715283 

5 

7 

46 

5.00698 

47 

15 

0.715283 

5 

10 

46 

7.152828 

47 

16 

0.71402 

11 

25 

3 

17.85049 

3 

16 

0.71402 : 

11 

22 

3 

15.70844 

3 

17 

0.710439 

31 

13 

28 

9.235707 

31 

17 

0.710439 

31 

14 

28 

9.946146 

31 

18 

0.709478 

36 

7 

46 

4.966347 

48 

18 

0.709478 

36 

10 

46 

7.094782 

48 

19 

0.710439 

31 

13 

28 

9.235707 

31 

19 

0.710439 

31 

14 

28 

9.946146 

31 

20 

0.709478 

36 

7 

46 

4.966347 

48 

20 

0.709478 

36 

10 

46 

7.094782 

48 

21 

0.710648 

27 

25 

3 

17.76619 

5 

21 

0.710648 

27 

22 

3 

15.63425 

5 

22 

0.713594 

13 

13 

28 

9.276717 

29 

22 

0.713594  * 

13 

14 

28 

9.99031 1r 

29 

23 

0.710648 

27 

25 

3 

17.76619 

5 

23 

0.710648 

27 

22 

3 

15.63425: 

5 

24 

0.713594 

13 

13 

28 

9.276717 

29 

24 

0.713594 

13 

14 

28 

9.99031 1 : 

29 

25 

0.715222 

6 

13.6 

21 

9.727022 

21 

25 

0.715222 

6 

14.4 

21 

10.2992 

21 

26 

0.716797 

1 

12.4 

37 

8.88828 

37 

26 

0.716797 

1 

13.6 

37 

9.748436 

37 

27 

0.715707 

3 

13 

28 

9.304187 

28 

27 

0.71 5707  f 

3 

14 

28 

10.01989 

28 

28 

0.711992 

23 

19.6 

9 

13.95505 

9 

28 

0.711992  ?'■ 

23 

18.4 

9 

13.10066 

9 

29 

0.712621 

20 

18.4 

13 

13.11222 

14 

29 

0.712621 & 

20 

17.6 

13 

12.54212 

14 

30 

0.712379 

22 

19 

11 

13.53519 

11 

30 

0.712379 . 

22 

18 

11 

12.82281 

11 

31 

0.70073 

50 

13.6 

21 

9.529924 

26 

31 

0.70073 

50 

14.4' 

21 

10.09051 

26 

32 

0.710606 

29 

12.4 

37 

8.811518 

43 

32 

0.710606 

29 

13.6 

37 

9.664246 

43 

33 

0.70368 

45 

13 

28 

9.147841 

36 

33 

0.70368; 

45 

14 

28 

9.851521: 

36 

34 

0.71145 

24 

16 

17 

11.3832 

17 

34 

0.71145 

24 

16 

17 

11.3832 - 

.17; ) 

35 

0.704337 

44 

16 

17 

11.2694 

18 

35 

0.704337 1 

44 

16 

17 

11.2694 

18 

36 

0.708184 

40 

13 

28 

9.206388; 

34 

36 

0.708184 

40 

14 

28 

9.914571 ! 

34 

37 

0.704945 

43 

16.6 

15 

11.70208 

15 

37 

0.704945 

43 

16.4 

15 

11.56109^ 

15 

38 

0.701729 

49 

16.6 

15 

11.64871 

16 

38 

0.701729 

49 

16.4 

15 

11.50836; 

16 

39 

0.700554 

51 

13.6 

21 

9.52754 

27 

39 

0.700554 

51 

14.4 

21 

10.08798 

27 

40 

0.71521 

7 

15.4 

19 

11.01424 

19 

40 

0.71521 

7 

15.6 

19 

11.15728 

19 

41 

0.710439 

33 

15.4 

19 

10.94076 

20 

41 

0.710439 

33 

15.6 

19 

11.08285: 

20 

42 

0.712593 

21 

12.4 

37 

8.836152 

41 

42 

0.712593 

21 

13.6 

37 

9.691264 

41 

43 

0.705602 

41 

13 

28 

9.172822 

35 

43 

0.705602 

41 

14 

28 

9.878424 

35 

44 

0.70902 

39 

13 

28 

9.217258 

33 

44 

0.70902 

39 

14 

28 

9.926278 

33 

45 

0.705602 

41 

19 

11 

13.40643 

12 

45 

0.705602 

41 

18 

11 

12.70083 

12 

46 

0.702531 

46 

13.6 

21 

9.554419 

24 

46 

0.702531 ; 

46 

14.4 

21 

10,11644!; 

24 

47 

0.702531 

46 

13.6 

21 

9.554419 

24 

47 

0.702531  ■ 

46 

144 

21 

10.11644 

•  24 

48 

0.702531 

46 

19.6 

9 

13.7696 

10 

48 

0.702531 

46 

18.4 

9 

12.92657 

10 

49 

0.712772 

17 

12.4 

37 

8.838375 

39 

49 

0.712772 

17 

13.6 

37 

9.693702! 

39 

50 

0.712772 

17 

12.4 

37 

8.838375 

39 

50 

0.712772 

17 

13.6 

37 

9.693702 : 

39 

51 

0.712772 

17 

18.4 

13 

13.11501 

13 

51 

0.712772 

17 

17.6 

13 

12.54479 
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(suitability) 

(suitability) 
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Extrin  @  | 

Intrin  @  [ 

Multiplic.  [ 
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Extrin  @  |_ 

Intrin  @  | 

Multiplic.  |_ 

Typical 

Rank 

.6  and  .4 

Rank 

Result 

Rank  | 

Typical 

Rank 

.51  &  .49  Rank 

Result 

Rank  | 

1 

0.713534 

15 

19.2 

1 

13.69985 

1 

1 

0.713534 

15 

16.32  1  ' 

11.64487 

1 

2 

0.714978 

8 

13.2 

44 

9.437715 

44 

2 

0.714978 

8 

15.72  44 

11.23946 

38 

3 

0.71603 

2 

12.8 

50 

9.165182 

50 

3 

0.71603 

2 

15.68  50 

11.22735 

41 

4 

0.714309 

9 

18.8 

7 

13.42902 

7 

4 

0.714309 

9 

16.28  7 

11.62896 

4 

5 

0.710151 

34 

15.2 

21 

10.7943 

23 

5 

0.710151 

34 

15.92  21 

11.30561 

28 

6 

0.70924 

38 

13.2 

44 

9.361965 

45 

6 

0.70924 

38 

15.72  44 

11.14925 

48 

7 

0.711153 

25 

14.8 

37 

10.52506 

42 

7 

0.711153 

25 

15.88  37 

11.2931 

32 

8 

0.709751 

35 

12.8 

50 

9.084811 

51 

8 

0.709751 

35 

15.68  50 

11.12889 

51 

9 

0.710476 

30 

19.2 

1 

13.64113 

2 

9 

0.710476 

30 

16.32  1 

11.59496 

5 

10 

0.712788 

16 

15.2 

21 

10.83438 

22 

10 

0.712788 

16 

15.92  21 

11.34758 

20 

11 

0.711142 

26 

18.8 

7 

13.36946 

8 

11 

0.711142 

26 

16.28  7 

11.57739 

8 

12 

0.714035 

10 

14.8 

37 

10.56772 

37 

12 

0.714035 

10 

15.88  37 

11.33888 

23 

13 

0.713743 

12 

19 

3 

13.56112 

4 

13 

0.713743 

12 

16.3  3 

11.63402 

3 

14 

0.715651 

4 

13 

46 

9.303466 

46 

14 

0.715651 

4 

15.7  46 

11.23572 

39 

15 

0.715283 

5 

13 

46 

9.298677 

47 

15 

0.715283 

5 

15.7  46 

11.22994 

i-  40 

16 

0.71402 

11 

19 

3 

13.56638 

3 

16 

0.71402 

11 

16.3  3 

11.63852 

17 

0.710439 

31 

15 

28 

10.65658 

29 

17 

0.710439 

31 

15.9  28 

11.29598 

29 

18 

0.709478 

36 

13 

46 

9.223216 

48 

18 

0.709478 

36 

15.7  46 

11.13881 

49 

19 

0.710439 

31 

15 

28 

10.65658 

29 

19 

0.710439 

31 

15.9  28 

11.29598 

29 

20 

0.709478 

36 

13 

46 

9.223216 

48 

20 

0.709478 

36 

15.7  46 

H.13881 

49 

21 

0.710648 

27 

19 

3 

13.50231 

5 

21 

0.710648 

27 

16.3  3 

11.58356 

6 

22 

0.713594 

13 

15 

28 

10.7039 

25 

22 

0.713594 

13 

15.9  ^’28 

11.34614 

21 

23 

0.710648 

27 

19 

3 

13.50231 

5 

23 

0.710648 

27 

16.3  3 

11.58356 

6 

24 

0.713594 

13 

15 

28 

10.7039 

25 

24 

0.713594 

13 

15.9  28 

11.34614 

21 

25 

0.715222 

6 

15.2 

21 

10.87138 

21 

25 

0.715222 

6 

15.92  21 

11.38634 

14 

26 

0.71 6797  ;j 

1 

14.8 

37 

10.60859 

35 

26 

0.716797 

1 

15.88  37 

11.38273 

16 

27 

0.715707 

3 

15 

28 

10.7356 

24 

27 

0.715707 

3 

15.9  28 

11.37974 

17 

28 

0.711992 

23 

17.2 

9 

12.24627 

9' 

28 

0.711992 

23 

16.12  9 

11.47732 

9 

29 

0.712621 

20 

16.8 

13 

11.97203 

14 

29 

0.712621- 

20 

16.08;;  13 

11.45894 

12 

30 

0.712379 

22 

17 

11 

12.11044 

10 

30 

0.71 2379 ; 

22 

16.1  ^.11 

11.46929 

10 

31 

0.70073 

50 

15.2 

21 

10.65109 

31 

31 

0.70073 

50 

15.92  21 

11.15562 

46 

32 

0.710606 

29 

14.8 

37 

10.51697 

43 

32 

0.710606 

29 

15.88  37 

11.28443 

33 

33 

0.70368 

45 

15; 

28 

10.5552 

38 

33 

0.70368 

45 

15.9  28 

1 1.18851 1; 

43 

34 

0.71145 

24 

16 

17 

11.3832 

16 

34 

0.71145 

24 

16  17 

11.3832 

15 

35 

0.704337 

44 

16 

17 

11. 2694 ji 

19 

35 

0.704337 

44 

16  17 

11.2694 

35 

36 

0.708184 

40 

15 

28 

10.62276 

34 

36 

0.708184 

40 

15.9  28 

11.26012 

36 

37 

0.704945 

43 

16.2 

15 

11.42011 

15 

37 

0.704945 

43 

16.02  15 

11.29322 

31 

38 

0.701729 

49 

16.2 

15 

11.36802 

17 

38 

0.701729 

49 

16.02  15 

1 1.24171 

37 

39 

0.700554 

51 

15.2 

21 

10.64843 

32 

39 

0.700554 

51 

15.92  21 

11.15283 

47 

40 

0.71521 

7 

15.8 

19 

11.30033 

18 

40 

0.71521 

7 

15.98  19 

11.42906; 

13 

41 

0.710439 

33 

15.8 

19 

11.22494 

20 

41 

0.710439 

33 

15.98  19 

11.35281 

19 

42 

0.712593 

21 

14.8 

37 

10.54638 

41 

42 

0.712593 

21 

15.88  37 

11.31598 

27 

43 

0.705602 

41 

15 

28 

10.58403 

36 

43 

0.705602 

41 

15.9  28 

11.21907 

42 

44 

0.70902 

39 

15 

28 

10.6353 

33 

44 

0.70902 

39 

15.9-i  28 

11.27342 

34 

45 

0.705602 

41 

17 

11 

11.99523 

12 

45 

0.705602 

41 

16.1  11 

11.36019 

18 

46 

0.702531 

46 

15.2 

21 

10.67847 

27 

46 

0.702531 

46 

15.92  21 

11.18429 

44 

47 

0.702531 

46 

15.2 

21 

10.67847 

27 

47 

0.702531 

46 

15.92  21 

1 1.1 8429  K' 

44 

48 

0.702531 

46 

17.2 

9 

12.08353 

11 

48 

0.702531 

46 

16.12  9 

11.3248 

24 

49 

0.712772 

17 

14.8 

37 

10.54903 

39 

49 

0.712772 

17 

15.88  37 

11.31882 

25 

50 

0.712772 

17 

14.8 

37 

10.54903 

39 

50 

0.712772 

17 

15.88  37 

11.31882 

25 

51 

0.712772 

17 

16.8 

13 

1 1 .97457 

13 

51 

0.712772 

17 

16.08  13 

11.46138 

11 

188 


(suitability)  _  ^  (suitability) 


Extrin  @  1 

Intrin  @  1 

Multipiic.  | 

Extrin  @  | 

Intrin  @  j 

Multipiic.  | 

MIX 

.9  and  .1 

Rank 

Typical 

Rank 

Result 

Rank  I 

MIX 

.8  and  .2  Rank 

Typical 

Rank 

Result 

Rank  j 

1 

0.49235 

41 

25.6 

1 

12.60417 

18 

1 

0.506263  41 

25.6 

1 

12.96033 

13 

2 

0.653574 

18 

12.6 

48 

8.235027 

45 

2 

0.628016  18 

12.6 

48 

7.913 

46 

3 

0.613193 

30 

11.8 

51 

7.235676 

48 

3 

0.597701  30 

11.8 

51 

7.052869 

51 

4 

0.451663 

47 

24.8 

4 

11.20125 

29 

4 

0.476315  47 

24.8 

4 

11.81262 

22 

5 

0.819733 

1 

21.6 

9 

17.70623 

1 

5 

0.755414  1 

21.6 

9 

16.31695 

1 

6 

0.653574 

18 

16.6 

39 

10.84932 

35 

6 

0.628016  18 

16.6 

39 

10.42506 

36 

7 

0.78455 

13 

20.8 

14 

16.31863 

4 

7 

0.72688  13 

20.8 

14 

15.11911 

4 

8 

0.613193 

30 

15.8 

43 

9.688448 

44 

8 

0.597701  30 

15.8 

43 

9.443672 

44 

9 

0.49235 

41 

23.6 

5 

11.61947 

25 

9 

0.506263  41 

23.6 

5 

11.9478 

20 

10 

0.819733 

1 

15.6 

44 

12.78783 

16 

10 

0.755414  1 

15.6 

44 

11.78446 

23  ; 

11 

0.451663 

47 

22.8 

8 

10.29792 

37 

11 

0.476315  47 

22.8 

8 

10.85999 

32 

12 

0.78455 

13 

14.8 

47 

11.61133 

26 

12 

0.72688  13 

14.8 

47 

10.75783 

33 

13 

0.478048 

43 

25.2 

2 

12.04681 

22 

13 

0.495807  43 

25.2 

2 

12.49435 

16 

14 

0.626065 

26 

12.2 

49 

7.637999 

47 

14 

0.607488  26 

12.2 

49 

7.411354 

49 

15 

0.639323 

23 

12.2 

49 

7.799737 

46 

15 

0.617526  23 

12.2 

49 

7.533814 

48 

16 

0.464825 

46 

25.2 

2 

11.71359 

24 

16 

0.485907  46 

25.2 

2 

12.24487 

18 

17 

0.807273 

6 

21.2 

11 

17.11418 

2 

17 

0.745687  6 

21. 2  i 

11 

15.80857 

2 

18 

0.626065 

26 

16.2 

41 

10.14226 

39 

18 

0.607488  26 

16.2 

41 

9.841306 

40 

19 

0.807273 

6 

21.2 

11 

17.11418 

2 

19 

0.745687  6 

21.2 

11 

15.80857 

2 

20 

0.626065 

26 

16.2 

41 

10.14226 

39 

20 

0.607488  26 

16.2 

41 

9.841306 

40 

21 

0.478048 

43 

23.2 

6 

11.09071  . 

31 

21 

0.495807  43 

23.2 

6 

11.50273; 

24 

22 

0.795755' 

10 

15.2 

45 

12.09547: 

20 

22 

0.736306  10 

15.2 

45 

11.19185 

28 

23 

0.478048 

43 

23.2 

6 

11.09071 

31 

23 

0.495807  43 

23.2 

6 

11.50273 

24 

24 

0.795755 

10 

15.2 

45 

12.09547 

20 

24 

0.736306  10 

15.2 

45 

11.19185! 

28 

25 

0.819733 

1 

17.6 

34 

14.4273 

9 

25 

0.755414  1 

17.6 

34 

13.29529 

11  : 

26 

0.78455 

13 

16.8 

37 

13.18043 

15 

26 

0.72688  15 

16.8 

37 

12.21159 

19 

27 

0.807273 

8 

17.2 

35 

13.88509 

13 

27 

0.745687  6 

17.2 

35 

12.82582 

V:"15'":: 

28 

0.348434 

49 

20.6 

16 

7.177747; 

49 

28 

0.39244  49 

20.6^ 

16 

8.08427 

45 

29 

0.315672 

51 

19.8 

19 

6.250296 

51 

29 

0.365731  51 

19.8 

19 

7.241467! 

50 

30 

0.325735 

50 

20.2 

17 

6.579837 

50 

30 

0.374094  50 

20.2  £ 
19.6' 

17 

7.5567 

!:;:47 

31 

0.80881 

4 

19.6 

21 

15.85268 

5 

31 

0.747397  4 

21 

14.64899 

'.5'  ■  'i 

32 

0.78455 

13 

18.8 

26 

14.74953 

8 

32 

0.72688  15 

18.8 

26 

13.66535 

8 

33 

0.799638 

9 

19.2 

23 

15.35305 

6 

33 

0.740096  9 

19.2? 

23 

14.20985 

6 

34 

0.542366 

38 

18.7 

27 

10.14224 

41 

34 

0.54412  38 

18.7 

27 

10.17504 

38 

35 

0.554912 

37 

19.7; 

20 

10.93176 

33 

35 

0.553233  37 

19.7 

20 

10.89869 

31  ; 

36 

0.791744 

12 

18.2 

32 

14.40975 

10 

36 

0.733377  12 

18.2^ 

32 

13  34746 

10 

37 

0.568733 

35 

19.1 

24 

10.8628 

34 

37 

0.563006  35 

191  . 

20.1 :: 

24 

10.75341 1- 

34 

38 

0.568733 

35 

20.1 

18 

11.43153 

27 

38 

0.563006  35 

18 

11.31642 

27 

39 

0.80881 

4 

18.6 

29 

15.04387 

7 

39 

0.747397  4 

18.6 

29 

13.90159 

7 

40 

0.53048 

39 

18.3 

30 

9.707785 

43 

40 

0.535424  39 

18.3 

30 

9.798257 

42 

41 

0.53048 

39 

19.3 

22 

10.23826 

38 

41 

0.535424  39 

19.3 

22 

10.33368 

37 

42 

0.78455 

13 

17.8 

33 

13.96498 

12 

42 

0.72688  15 

17.8 

33 

12.93847 

14 

43 

0.636165 

24 

18.7 

27 

11.89629 

23 

43 

0.614627  24 

18.7 

27 

11.49353 

26 

44 

0.624431 

29 

16.7 

38 

10.428 

36 

44 

0.605988  29 

16.7 

38 

10.12001 

39 

45 

0.636165 

24 

21.2 

11 

13.4867 

14 

45 

0.614627  24 

21.2 

11 

13.0301  j: 

12 

46 

0.648993 

20 

19.1 

24 

12.39576 

19 

46 

0.623809  20 

19.1 

24 

11.91475 

21 

47 

0.648993 

20 

17.1 

36 

11.09778 

30 

47 

0.623809  20 

17.1 

36 

10.66714 

35 

48 

0.648993 

20 

21.6 

9 

14.01825 

11 

48 

0.623809  20 

21.6 

9 

13.47428 

9 

49 

0.613193 

30 

18.3 

30 

11.22143 

28 

49 

0.597701  30 

18.3 

30 

10.93792 

30 

50 

0.613193 

30 

16.3 

40 

9.995044 

42 

50 

0.597701  30 

1 6.3  r 

40 

9.742522; 

43 

51 

0.613193 

30 

20.8 

14 

12.75441 

17 

51 

0.597701  30 

20.8 

14 

12.43218 

17 

189 


(suitability) 

MIX  | 

Extrin  @  [ 

Rank  | 

Intrin  @  [_ 

Rank  | 

Multiplic.  [ 

.7  and  .3 

Typical 

Result 

Rank  | 

1 

0.520554 

41 

25.6 

1 

13.32618 

6 

2 

0.602426 

18 

12.6 

48 

7.59057 

48 

3 

0.581509 

30 

11.8 

51 

6.861801 

51 

4 

0.500702 

47 

24.8 

4 

12.41741 

14 

5 

0.68704 

1 

21.6 

9 

14.84007 

1 

6 

0.602426 

18 

16.6 

39 

10.00027 

37 

7 

0.664659 

16 

20.8 

14 

13.82491 

4 

8 

0.581509 

30 

15.8 

43 

9.187835 

44 

9 

0.520554 

41 

23.6 

5 

12.28507 

15 

10 

0.68704 

1 

15.6 

44 

10.71783 

29 

11 

0.500702 

47 

22.8 

8 

11.41601 

24 

12 

0.664659 

16 

14.8 

47 

9.836958 

39 

13 

0.513772 

43 

25.2 

2 

12.94705 

8 

14 

0.588415 

26 

12.2 

49 

7.178661 

50 

15 

0.595484 

23 

12.2 

49 

7.2649 

49 

16 

0.507162 

46 

25.2 

2 

12.78049 

10 

17 

0.679619 

6 

21.2 

11 

14.40793 

2 

18 

0.588415 

26 

16.2 

41 

9.53232 

41 

19 

0.679619 

6 

21.2 1 

11 

14.40793 

2 

20 

0.588415 

26 

16.2 

41 

9.53232 

41 

21 

0.513772 

43 

23.2 

6 

11.9195 

19 

22 

0.672173 

10 

15.2 

45 

10.21702 

34 

23 

0.513772 

43 

23.2 

6 

11.9195 

19 

24 

0.672173 

10 

15.2 

45 

10.21702 

34 

25 

0.68704 

1 

17.6: 

34 

12.09191 

18 

26 

0.664659 

13 

16.8* 

37 

11.16628 

26 

27 

0.679619 

6 

17.2 

35 

11.68945 

22 

28 

0.442299 

49 

20.6 

16 

9.11135 

45 

29 

0.422002- 

51 

19.8 

19 

8.355647 

47 

30 

0.42863 

50 

20.2 

17 

8.65832 

46 

31 

0.681652 

4 

19.6 

21 

13.36039 

5 

32 

0.664659 

13 

18.8 

26 

12.4956 

13 

33 

0.675903 

9 

19.2 

23 

12.97733 

7 

34 

0.545882 

38 

18.7 

27 

10.20799 

36 

35 

0.551567 

37 

19.7 

20 

10.86587 

28 

36 

0.670248 

12 

18.2 

32 

12.1985 

16 

37 

0.557503 

35 

19.1 

24 

10.64832 

30 

38 

0.557503 

35 

20.1 

18 

11.20582 

25 

39 

0.681652 

4 

18.6 

29 

12.67874 

11 

40 

0.54043 

39 

18.3 

30 

9.88986 

38 

41 

0.54043 

39 

19.3 

22 

10.43029 

32 

42 

0.664659 

13 

17.8 

33 

1 1 .83094 

21 

43 

0.592831 

24 

18.7 

27 

11.08594 

27 

44 

0.587043 

29 

16.7 

38 

9.803618 

40 

45 

0.592831 

24 

21.2 

11 

12.56801 

12 

46 

0.598575 

20 

19.1 

24 

11.43278 

23 

47 

0.598575 

20 

17.1 

36 

10.23563 

33 

48 

0.598575 

20 

21.6 

9 

12.92921 

9 

49 

0.581509 

30 

18.3 

30 

10.64161 

31 

50 

0.581509 

30 

16.3 

40 

9.47859 

43 

51 

0.581509 

30 

20.8 

14 

12.09538 

17 

(suitability) 


MIX 

l  Extrin  @  | 

I  Intrin  @  |  i 

|  Multiplic.  | 

|  .6  and  .4 

Rank  | 

|  Typical 

Rank  | 

Result 

Rank  | 

1 

0.535307 

41 

25.6 

1 

13.70386 

1 

2 

0.576024 

18 

12.6 

48 

7.257908 

48 

3 

0.565757 

30 

11.8 

51 

6.675936 

51 

4 

0.5253 

47 

24.8 

4 

13.02744 

5 

5 

0.617574 

1 

21.6 

9 

13.33961 

3 

6 

0.576024 

18 

16.6 

39 

9.562006 

39 

7 

0.607589 

16 

20.8 

14 

12.63785  : 

8 

8 

0.565757 

30 

15.8 

43 

8.938965 

47 

9 

0.535307 

41 

23.6 

5 

12.63325 

9 

10 

0.617574 

1 

15.6 

44 

9.634159 

37 

11 

0.5253 

47 

22.8 

8 

11.97684 

15 

12 

0.607589 

16 

14.8 

47 

8.992313 

46 

13 

0.531929 

43 

25.2 

2 

13.40462 

2 

14 

0.569136 

26 

12.2 

49 

6.943456 

50 

15 

0.572557 

20 

12.2 

49 

6.985197 

49 

16 

0.528595 

46 

25.2 

2 

13.32059 

4 

17 

0.614222 

4 

2 1.2 1' 

11 

13.0215 

6 

18 

0.569136 

26 

16.2 

41 

9.219999 

44 

19 

0.614222 

4 

21.2?" 

11 

13.0215 

6 

20 

0.569136 

26 

16.2 

41 

9.219999 

44 

21 

0.531929 

43 

23.2 

6 

12.34076 

11 

22 

0.610899 

10 

15.2 

45 

9.285658 

41 

23 

0.531929 

43 

23.2 

6 

12.34076 

11 

24 

0.610899 

10 

15.2 

45 

9.285658 

41 

25 

0.617574 

1 

17.6?; 

34 

10.86931 

23 

26 

0.607589 

13 

16.8 

37 

10.20749 

33 

27 

0.614222 

4 

17.2ij 

35 

10.56461 

27 

28 

0.495856 

49 

20.6?* 

16 

10.21463 

32 

29 

0.486008 1 

51 

19.8 

19 

9.622962  ? 

38 

30 

0.489252 

50 

20.2  ;• 

17 

9.8829 

35 

31 

0.613368 

7 

19.6 

21 

12.022 

14 

32 

0.607589 

13 

18.8 

26 

11.42267 

18 

33 

0.611323 

9 

19.2 

23 

11.7374 

17 

34 

0.547553 

38 

18.7 

27 

10.23924 

31 

35 

0.549864 

37 

19.7 

20 

10.83232  !;- 

24 

36 

0.609399 

12 

18.2 

32 

11.09106 

21 

37 

0.552286 

35 

19.1 

24 

10.54867 

28 

38 

0.552286 

35 

20.1 

18 

11.10095 

20 

39 

0.613368 

7 

18.6 

29 

11.40864 

19 

40 

0.545356 

39 

18.3 

30 

9.98002 

34 

41 

0.545356 

39 

19.3 

22 

10.52538 

29 

42 

0.607589 

13 

17.8 

33 

10.81508 

25 

43 

0.570138 

24 

18.7 

27 

10.66158  ? 

26 

44 

0.567885 

29 

16.7 

38 

9.483682 

40 

45 

0.570138 

24 

21.2 

11 

12.08692 

13 

46 

0.572511 

21 

19.1 

24 

10.93495 

22 

47 

0.57251 1 

21 

17.1 

36 

9.78993 

36 

48 

0.572511 

21 

21.6 

9 

12.36623 

10 

49 

0.565757 

30 

18.3 

30 

10.35336 

30 

50 

0.565757 

30 

16.3 

40 

9.221844 

43 

51 

0.565757 

30 

20.8 

14 

11.76775 

16 

190 


(suitability) 

(suitability) 

MIX 

Extrin  @  |_ 

Intrin  @  [ 

Multiplic.  |_ 

MIX 

Extrin  @  [_ 

Intrin  @  j_ 

Multiplic.  | 

.51  &  .49 

Rank 

Typical 

Rank 

Result 

Rank  | 

Typical 

Rank 

Typical 

Rank 

Result  Rank  | 

1 

0.548908 

38 

25.6 

1 

14.05204 

1 

1 

0.713534 

15 

25.6 

1 

18.26647  1 

2 

0.552966 

18 

12.6 

48 
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